{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 波士顿房价预测任务\n",
    "\n",
    "上一节我们初步认识了神经网络的基本概念（如神经元、多层连接、前向计算、计算图）和模型结构三要素（模型假设、评价函数和优化算法）。本节将以“波士顿房价预测”任务为例，向读者介绍使用Python语言和Numpy库来构建神经网络模型的思考过程和操作方法。\n",
    "\n",
    "波士顿房价预测是一个经典的机器学习任务，类似于程序员世界的“Hello World”。和大家对房价的普遍认知相同，波士顿地区的房价受诸多因素影响。该数据集统计了13种可能影响房价的因素和该类型房屋的均价，期望构建一个基于13个因素进行房价预测的模型，如 **图1** 所示。\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/abce0cb2a92f4e679c6855cfa520491597171533a0b0447e8d51d904446e213e\" width=\"500\" hegiht=\"\" ></center>\n",
    "<center><br>图1：波士顿房价影响因素示意图</br></center>\n",
    "<br></br>\n",
    "\n",
    "对于预测问题，可以根据预测输出的类型是连续的实数值，还是离散的标签，区分为回归任务和分类任务。因为房价是一个连续值，所以房价预测显然是一个回归任务。下面我们尝试用最简单的线性回归模型解决这个问题，并用神经网络来实现这个模型。\n",
    "\n",
    "## 线性回归模型\n",
    "\n",
    "假设房价和各影响因素之间能够用线性关系来描述：\n",
    "\n",
    "$$y = {\\sum_{j=1}^Mx_j w_j} + b$$\n",
    "\n",
    "模型的求解即是通过数据拟合出每个$w_j$和$b$。其中，$w_j$和$b$分别表示该线性模型的权重和偏置。一维情况下，$w_j$ 和 $b$ 是直线的斜率和截距。\n",
    "\n",
    "线性回归模型使用均方误差作为损失函数（Loss），用以衡量预测房价和真实房价的差异，公式如下：\n",
    "\n",
    "$$MSE = \\frac{1}{n} \\sum_{i=1}^n(\\hat{Y_i} - {Y_i})^{2}$$\n",
    "\n",
    "------\n",
    "**思考：**\n",
    "\n",
    "为什么要以均方误差作为损失函数？即将模型在每个训练样本上的预测误差加和，来衡量整体样本的准确性。这是因为损失函数的设计不仅仅要考虑“合理性”，同样需要考虑“易解性”，这个问题在后面的内容中会详细阐述。\n",
    "\n",
    "------\n",
    "\n",
    "## 线性回归模型的神经网络结构\n",
    "\n",
    "神经网络的标准结构中每个神经元由加权和与非线性变换构成，然后将多个神经元分层的摆放并连接形成神经网络。线性回归模型可以认为是神经网络模型的一种极简特例，是一个只有加权和、没有非线性变换的神经元（无需形成网络），如 **图2** 所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/f9117a5a34d44b1eab85147e62b4e6295e485e48d79d4a03adaa14a447ffd230\" width=\"300\" hegiht=\"\" ></center>\n",
    "<center><br>图2：线性回归模型的神经网络结构</br></center>\n",
    "<br></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 构建波士顿房价预测任务的神经网络模型\n",
    "\n",
    "深度学习不仅实现了模型的端到端学习，还推动了人工智能进入工业大生产阶段，产生了标准化、自动化和模块化的通用框架。不同场景的深度学习模型具备一定的通用性，五个步骤即可完成模型的构建和训练，如 **图3** 所示。\n",
    "<br></br>\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/12fdca24a3b94166a9e8c815ef4b0e4ddfec541f3a024a4392f1fc17fa186c7b\" width=\"800\" hegiht=\"\" ></center>\n",
    "<center><br>图3：构建神经网络/深度学习模型的基本步骤</br></center>\n",
    "<br></br>\n",
    "\n",
    "正是由于深度学习的建模和训练的过程存在通用性，在构建不同的模型时，只有模型三要素不同，其它步骤基本一致，深度学习框架才有用武之地。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据处理\n",
    "\n",
    "数据处理包含五个部分：数据导入、数据形状变换、数据集划分、数据归一化处理和封装`load data`函数。数据预处理后，才能被模型调用。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "* 本教程中的代码都可以在AI Studio上直接运行，Print结果都是基于程序真实运行的结果。\n",
    "* 由于是真实案例，代码之间存在依赖关系，因此需要读者逐条、全部运行，否则会导致命令执行报错。\n",
    "\n",
    "------\n",
    "\n",
    "### 读入数据\n",
    "\n",
    "通过如下代码读入数据，了解下波士顿房价的数据集结构，数据存放在本地目录下housing.data文件中。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6.320e-03, 1.800e+01, 2.310e+00, ..., 3.969e+02, 7.880e+00,\n",
       "       1.190e+01])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 导入需要用到的package\n",
    "import numpy as np\n",
    "import json\n",
    "# 读入训练数据\n",
    "datafile = './work/housing.data'\n",
    "data = np.fromfile(datafile, sep=' ')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据形状变换\n",
    "\n",
    "由于读入的原始数据是1维的，所有数据都连在一起。因此需要我们将数据的形状进行变换，形成一个2维的矩阵，每行为一个数据样本（14个值），每个数据样本包含13个$X$（影响房价的特征）和一个$Y$（该类型房屋的均价）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 读入之后的数据被转化成1维array，其中array的第0-13项是第一条数据，第14-27项是第二条数据，以此类推.... \n",
    "# 这里对原始数据做reshape，变成N x 14的形式\n",
    "feature_names = [ 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE','DIS', \n",
    "                 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV' ]\n",
    "feature_num = len(feature_names)\n",
    "data = data.reshape([data.shape[0] // feature_num, feature_num])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14,)\n",
      "[6.320e-03 1.800e+01 2.310e+00 0.000e+00 5.380e-01 6.575e+00 6.520e+01\n",
      " 4.090e+00 1.000e+00 2.960e+02 1.530e+01 3.969e+02 4.980e+00 2.400e+01]\n"
     ]
    }
   ],
   "source": [
    "# 查看数据\n",
    "x = data[0]\n",
    "print(x.shape)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集划分\n",
    "\n",
    "将数据集划分成训练集和测试集，其中训练集用于确定模型的参数，测试集用于评判模型的效果。为什么要对数据集进行拆分，而不能直接应用于模型训练呢？这与学生时代的授课和考试关系比较类似，如 **图4** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/a1c845a50e28474d9aa72028edfea33f1a3deca1d54d40ec94ba366d3a18c408\" width=\"600\" hegiht=\"\" ></center>\n",
    "<center><br>图4：训练集和测试集拆分的意义</br></center>\n",
    "<br></br>\n",
    "\n",
    "上学时总有一些自作聪明的同学，平时不认真学习，考试前临阵抱佛脚，将习题死记硬背下来，但是成绩往往并不好。因为学校期望学生掌握的是知识，而不仅仅是习题本身。另出新的考题，才能鼓励学生努力去掌握习题背后的原理。同样我们期望模型学习的是任务的本质规律，而不是训练数据本身，模型训练未使用的数据，才能更真实的评估模型的效果。\n",
    "\n",
    "在本案例中，我们将80%的数据用作训练集，20%用作测试集，实现代码如下。通过打印训练集的形状，可以发现共有404个样本，每个样本含有13个特征和1个预测值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(404, 14)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratio = 0.8\n",
    "offset = int(data.shape[0] * ratio)\n",
    "training_data = data[:offset]\n",
    "training_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据归一化处理\n",
    "\n",
    "对每个特征进行归一化处理，使得每个特征的取值缩放到0~1之间。这样做有两个好处：一是模型训练更高效；二是特征前的权重大小可以代表该变量对预测结果的贡献度（因为每个特征值本身的范围相同）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 计算train数据集的最大值，最小值，平均值\n",
    "maximums, minimums, avgs = \\\n",
    "                     training_data.max(axis=0), \\\n",
    "                     training_data.min(axis=0), \\\n",
    "     training_data.sum(axis=0) / training_data.shape[0]\n",
    "# 对数据进行归一化处理\n",
    "for i in range(feature_num):\n",
    "    #print(maximums[i], minimums[i], avgs[i])\n",
    "    data[:, i] = (data[:, i] - minimums[i]) / (maximums[i] - minimums[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 封装成load data函数\n",
    "\n",
    "将上述几个数据处理操作封装成`load data`函数，以便下一步模型的调用，实现方法如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    # 从文件导入数据\n",
    "    datafile = './work/housing.data'\n",
    "    data = np.fromfile(datafile, sep=' ')\n",
    "\n",
    "    # 每条数据包括14项，其中前面13项是影响因素，第14项是相应的房屋价格中位数\n",
    "    feature_names = [ 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', \\\n",
    "                      'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV' ]\n",
    "    feature_num = len(feature_names)\n",
    "\n",
    "    # 将原始数据进行Reshape，变成[N, 14]这样的形状\n",
    "    data = data.reshape([data.shape[0] // feature_num, feature_num])\n",
    "\n",
    "    # 将原数据集拆分成训练集和测试集\n",
    "    # 这里使用80%的数据做训练，20%的数据做测试\n",
    "    # 测试集和训练集必须是没有交集的\n",
    "    ratio = 0.8\n",
    "    offset = int(data.shape[0] * ratio)\n",
    "    training_data = data[:offset]\n",
    "\n",
    "    # 计算训练集的最大值，最小值，平均值\n",
    "    maximums, minimums, avgs = training_data.max(axis=0), training_data.min(axis=0), \\\n",
    "                                 training_data.sum(axis=0) / training_data.shape[0]\n",
    "\n",
    "    # 对数据进行归一化处理\n",
    "    for i in range(feature_num):\n",
    "        #print(maximums[i], minimums[i], avgs[i])\n",
    "        data[:, i] = (data[:, i] - minimums[i]) / (maximums[i] - minimums[i])\n",
    "\n",
    "    # 训练集和测试集的划分比例\n",
    "    training_data = data[:offset]\n",
    "    test_data = data[offset:]\n",
    "    return training_data, test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取数据\n",
    "training_data, test_data = load_data()\n",
    "x = training_data[:, :-1]\n",
    "y = training_data[:, -1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.         0.18       0.07344184 0.         0.31481481 0.57750527\n",
      " 0.64160659 0.26920314 0.         0.22755741 0.28723404 1.\n",
      " 0.08967991]\n",
      "[0.42222222]\n"
     ]
    }
   ],
   "source": [
    "# 查看数据\n",
    "print(x[0])\n",
    "print(y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型设计\n",
    "\n",
    "模型设计是深度学习模型关键要素之一，也称为网络结构设计，相当于模型的假设空间，即实现模型“前向计算”（从输入到输出）的过程。\n",
    "\n",
    "如果将输入特征和输出预测值均以向量表示，输入特征$x$有13个分量，$y$有1个分量，那么参数权重的形状（shape）是$13\\times1$。假设我们以如下任意数字赋值参数做初始化：\n",
    "$$w=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, -0.1, -0.2, -0.3, -0.4, 0.0]$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, -0.1, -0.2, -0.3, -0.4, 0.0]\n",
    "w = np.array(w).reshape([13, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "取出第1条样本数据，观察样本的特征向量与参数向量相乘的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.69474855]\n"
     ]
    }
   ],
   "source": [
    "x1=x[0]\n",
    "t = np.dot(x1, w)\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "完整的线性回归公式，还需要初始化偏移量$b$，同样随意赋初值-0.2。那么，线性回归模型的完整输出是$z=t+b$，这个从特征和参数计算输出值的过程称为“前向计算”。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.49474855]\n"
     ]
    }
   ],
   "source": [
    "b = -0.2\n",
    "z = t + b\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上述计算预测输出的过程以“类和对象”的方式来描述，类成员变量有参数$w$和$b$。通过写一个`forward`函数（代表“前向计算”）完成上述从特征和参数到输出预测值的计算过程，代码如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，\n",
    "        # 此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "基于Network类的定义，模型的计算过程如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.39362982]\n"
     ]
    }
   ],
   "source": [
    "net = Network(13)\n",
    "x1 = x[0]\n",
    "y1 = y[0]\n",
    "z = net.forward(x1)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练配置\n",
    "\n",
    "模型设计完成后，需要通过训练配置寻找模型的最优值，即通过损失函数来衡量模型的好坏。训练配置也是深度学习模型关键要素之一。\n",
    "\n",
    "通过模型计算$x_1$表示的影响因素所对应的房价应该是$z$, 但实际数据告诉我们房价是$y$。这时我们需要有某种指标来衡量预测值$z$跟真实值$y$之间的差距。对于回归问题，最常采用的衡量方法是使用均方误差作为评价模型好坏的指标，具体定义如下：\n",
    "\n",
    "$$Loss = (y - z)^2$$\n",
    "\n",
    "上式中的$Loss$（简记为: $L$）通常也被称作损失函数，它是衡量模型好坏的指标。在回归问题中，均方误差是一种比较常见的形式，分类问题中通常会采用交叉熵作为损失函数，在后续的章节中会更详细的介绍。对一个样本计算损失函数值的实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3.88644793]\n"
     ]
    }
   ],
   "source": [
    "Loss = (y1 - z)*(y1 - z)\n",
    "print(Loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因为计算损失函数时需要把每个样本的损失函数值都考虑到，所以我们需要对单个样本的损失函数进行求和，并除以样本总数$N$。\n",
    "$$Loss= \\frac{1}{N}\\sum_{i=1}^N{(y_i - z_i)^2}$$\n",
    "在Network类下面添加损失函数的计算过程如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        cost = error * error\n",
    "        cost = np.mean(cost)\n",
    "        return cost\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用定义的Network类，可以方便的计算预测值和损失函数。需要注意的是，类中的变量$x$, $w$，$b$, $z$, $error$等均是向量。以变量$x$为例，共有两个维度，一个代表特征数量（值为13），一个代表样本数量，代码如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predict:  [[2.39362982]\n",
      " [2.46752393]\n",
      " [2.02483479]]\n",
      "loss: 3.384496992612791\n"
     ]
    }
   ],
   "source": [
    "net = Network(13)\n",
    "# 此处可以一次性计算多个样本的预测值和损失函数\n",
    "x1 = x[0:3]\n",
    "y1 = y[0:3]\n",
    "z = net.forward(x1)\n",
    "print('predict: ', z)\n",
    "loss = net.loss(z, y1)\n",
    "print('loss:', loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练过程\n",
    "\n",
    "上述计算过程描述了如何构建神经网络，通过神经网络完成预测值和损失函数的计算。接下来介绍如何求解参数$w$和$b$的数值，这个过程也称为模型训练过程。训练过程是深度学习模型的关键要素之一，其目标是让定义的损失函数$Loss$尽可能的小，也就是说找到一个参数解$w$和$b$，使得损失函数取得极小值。\n",
    "\n",
    "我们先做一个小测试：如 **图5** 所示，基于微积分知识，求一条曲线在某个点的斜率等于函数在该点的导数值。那么大家思考下，当处于曲线的极值点时，该点的斜率是多少？\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/94f0437e6a454a0682f3b831c96a62bdaf40898af25145ec9b5b50bc80391f5c\" width=\"300\" hegiht=\"\" ></center>\n",
    "<center><br>图5：曲线斜率等于导数值</br></center>\n",
    "<br></br>\n",
    "\n",
    "这个问题并不难回答，处于曲线极值点时的斜率为0，即函数在极值点的导数为0。那么，让损失函数取极小值的$w$和$b$应该是下述方程组的解：\n",
    "$$\\frac{\\partial{L}}{\\partial{w}}=0$$\n",
    "$$\\frac{\\partial{L}}{\\partial{b}}=0$$\n",
    "\n",
    "将样本数据$(x, y)$带入上面的方程组中即可求解出$w$和$b$的值，但是这种方法只对线性回归这样简单的任务有效。如果模型中含有非线性变换，或者损失函数不是均方差这种简单的形式，则很难通过上式求解。为了解决这个问题，下面我们将引入更加普适的数值求解方法：梯度下降法。\n",
    "\n",
    "### 梯度下降法\n",
    "\n",
    "在现实中存在大量的函数正向求解容易，但反向求解较难，被称为单向函数，这种函数在密码学中有大量的应用。密码锁的特点是可以迅速判断一个密钥是否是正确的(已知$x$，求$y$很容易)，但是即使获取到密码锁系统，无法破解出正确的密钥是什么（已知$y$，求$x$很难）。\n",
    "\n",
    "这种情况特别类似于一位想从山峰走到坡谷的盲人，他看不见坡谷在哪（无法逆向求解出$Loss$导数为0时的参数值），但可以伸脚探索身边的坡度（当前点的导数值，也称为梯度）。那么，求解Loss函数最小值可以这样实现：从当前的参数取值，一步步的按照下坡的方向下降，直到走到最低点。这种方法笔者称它为“盲人下坡法”。哦不，有个更正式的说法“梯度下降法”。\n",
    "\n",
    "训练的关键是找到一组$(w, b)$，使得损失函数$L$取极小值。我们先看一下损失函数$L$只随两个参数$w_5$、$w_9$变化时的简单情形，启发下寻解的思路。\n",
    "$$L=L(w_5, w_9)$$\n",
    "这里我们将$w_0, w_1, ..., w_{12}$中除$w_5, w_9$之外的参数和$b$都固定下来，可以用图画出$L(w_5, w_9)$的形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "net = Network(13)\n",
    "losses = []\n",
    "#只画出参数w5和w9在区间[-160, 160]的曲线部分，以及包含损失函数的极值\n",
    "w5 = np.arange(-160.0, 160.0, 1.0)\n",
    "w9 = np.arange(-160.0, 160.0, 1.0)\n",
    "losses = np.zeros([len(w5), len(w9)])\n",
    "\n",
    "#计算设定区域内每个参数取值所对应的Loss\n",
    "for i in range(len(w5)):\n",
    "    for j in range(len(w9)):\n",
    "        net.w[5] = w5[i]\n",
    "        net.w[9] = w9[j]\n",
    "        z = net.forward(x)\n",
    "        loss = net.loss(z, y)\n",
    "        losses[i, j] = loss\n",
    "\n",
    "#使用matplotlib将两个变量和对应的Loss作3D图\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "fig = plt.figure()\n",
    "ax = Axes3D(fig)\n",
    "\n",
    "w5, w9 = np.meshgrid(w5, w9)\n",
    "\n",
    "ax.plot_surface(w5, w9, losses, rstride=1, cstride=1, cmap='rainbow')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对于这种简单情形，我们利用上面的程序，可以在三维空间中画出损失函数随参数变化的曲面图。从图中可以看出有些区域的函数值明显比周围的点小。\n",
    "\n",
    "需要说明的是：为什么这里我们选择$w_5$和$w_9$来画图？这是因为选择这两个参数的时候，可比较直观的从损失函数的曲面图上发现极值点的存在。其他参数组合，从图形上观测损失函数的极值点不够直观。\n",
    "\n",
    "观察上述曲线呈现出“圆滑”的坡度，这正是我们选择以均方误差作为损失函数的原因之一。**图6** 呈现了只有一个参数维度时，均方误差和绝对值误差（只将每个样本的误差累加，不做平方处理）的损失函数曲线图。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/99487dca6520441db5073d1c154b5d2fb1174b5cf4d946c29f9d80a209bc2687\" width=\"700\" hegiht=\"40\" ></center>\n",
    "<center><br>图6：均方误差和绝对值误差损失函数曲线图</br></center>\n",
    "<br></br>\n",
    "\n",
    "由此可见，均方误差表现的“圆滑”的坡度有两个好处：\n",
    "\n",
    "* 曲线的最低点是可导的。\n",
    "* 越接近最低点，曲线的坡度逐渐放缓，有助于通过当前的梯度来判断接近最低点的程度（是否逐渐减少步长，以免错过最低点）。\n",
    "\n",
    "而绝对值误差是不具备这两个特性的，这也是损失函数的设计不仅仅要考虑“合理性”，还要追求“易解性”的原因。\n",
    "\n",
    "现在我们要找出一组$[w_5, w_9]$的值，使得损失函数最小，实现梯度下降法的方案如下：\n",
    "\n",
    "- 步骤1：随机的选一组初始值，例如：$[w_5, w_9] = [-100.0, -100.0]$\n",
    "- 步骤2：选取下一个点$[w_5^{'} , w_9^{'}]$，使得$L(w_5^{'} , w_9^{'}) < L(w_5, w_9)$\n",
    "- 步骤3：重复步骤2，直到损失函数几乎不再下降。\n",
    "\n",
    "如何选择$[w_5^{'} , w_9^{'}]$是至关重要的，第一要保证$L$是下降的，第二要使得下降的趋势尽可能的快。微积分的基础知识告诉我们，沿着梯度的反方向，是函数值下降最快的方向，如 **图7** 所示。简单理解，函数在某一个点的梯度方向是曲线斜率最大的方向，但梯度方向是向上的，所以下降最快的是梯度的反方向。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/5f8322f6172542dab0f78684b70efe45d819895332af4cabb7c536217ab0bb26\" width=\"400\" hegiht=\"40\" ></center>\n",
    "<center><br>图7：梯度下降方向示意图</br></center>\n",
    "<br></br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 计算梯度\n",
    "\n",
    "上面我们讲过了损失函数的计算方法，这里稍微改写，为了使梯度计算更加简洁，引入因子$\\frac{1}{2}$，定义损失函数如下：\n",
    "\n",
    "$$L= \\frac{1}{2N}\\sum_{i=1}^N{(y_i - z_i)^2}$$\n",
    "\n",
    "其中$z_i$是网络对第$i$个样本的预测值：\n",
    "\n",
    "$$z_i = \\sum_{j=0}^{12}{x_i^{j}\\cdot w_j} + b$$\n",
    "\n",
    "梯度的定义：\n",
    "\n",
    "$$𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡 = (\\frac{\\partial{L}}{\\partial{w_0}},\\frac{\\partial{L}}{\\partial{w_1}}, ... ,\\frac{\\partial{L}}{\\partial{w_{12}}} ,\\frac{\\partial{L}}{\\partial{b}})$$\n",
    "\n",
    "可以计算出$L$对$w$和$b$的偏导数：\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)\\frac{\\partial{z_i}}{\\partial{w_j}}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)x_i^{j}}$$\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{b}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)\\frac{\\partial{z_i}}{\\partial{b}}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)}$$\n",
    "\n",
    "从导数的计算过程可以看出，因子$\\frac{1}{2}$被消掉了，这是因为二次函数求导的时候会产生因子$2$，这也是我们将损失函数改写的原因。\n",
    "\n",
    "下面我们考虑只有一个样本的情况下，计算梯度：\n",
    "\n",
    "$$L= \\frac{1}{2}{(y_i - z_i)^2}$$\n",
    "\n",
    "$$z_1 = {x_1^{0}\\cdot w_0} + {x_1^{1}\\cdot w_1} + ...  + {x_1^{12}\\cdot w_{12}} + b$$\n",
    "\n",
    "可以计算出：\n",
    "\n",
    "$$L= \\frac{1}{2}{({x_1^{0}\\cdot w_0} + {x_1^{1}\\cdot w_1} + ...  + {x_1^{12}\\cdot w_{12}} + b - y_1)^2}$$\n",
    "\n",
    "可以计算出$L$对$w$和$b$的偏导数：\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{w_0}} = ({x_1^{0}\\cdot w_0} + {x_1^{1}\\cdot w_1} + ...  + {x_1^{12}\\cdot w_12} + b - y_1)\\cdot x_1^{0}=({z_1} - {y_1})\\cdot x_1^{0}$$\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{b}} = ({x_1^{0}\\cdot w_0} + {x_1^{1}\\cdot w_1} + ...  + {x_1^{12}\\cdot w_{12}} + b - y_1)\\cdot 1 = ({z_1} - {y_1})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "可以通过具体的程序查看每个变量的数据和维度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1 [0.         0.18       0.07344184 0.         0.31481481 0.57750527\n",
      " 0.64160659 0.26920314 0.         0.22755741 0.28723404 1.\n",
      " 0.08967991], shape (13,)\n",
      "y1 [0.42222222], shape (1,)\n",
      "z1 [130.86954441], shape (1,)\n"
     ]
    }
   ],
   "source": [
    "x1 = x[0]\n",
    "y1 = y[0]\n",
    "z1 = net.forward(x1)\n",
    "print('x1 {}, shape {}'.format(x1, x1.shape))\n",
    "print('y1 {}, shape {}'.format(y1, y1.shape))\n",
    "print('z1 {}, shape {}'.format(z1, z1.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "按上面的公式，当只有一个样本时，可以计算某个$w_j$，比如$w_0$的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w0 [0.]\n"
     ]
    }
   ],
   "source": [
    "gradient_w0 = (z1 - y1) * x1[0]\n",
    "print('gradient_w0 {}'.format(gradient_w0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "同样我们可以计算$w_1$的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w1 [23.48051799]\n"
     ]
    }
   ],
   "source": [
    "gradient_w1 = (z1 - y1) * x1[1]\n",
    "print('gradient_w1 {}'.format(gradient_w1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "依次计算$w_2$的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w1 [9.58029163]\n"
     ]
    }
   ],
   "source": [
    "gradient_w2= (z1 - y1) * x1[2]\n",
    "print('gradient_w1 {}'.format(gradient_w2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "聪明的读者可能已经想到，写一个for循环即可计算从$w_0$到$w_{12}$的所有权重的梯度，该方法读者可以自行实现。\n",
    "\n",
    "### 使用Numpy进行梯度计算\n",
    "\n",
    "基于Numpy广播机制（对向量和矩阵计算如同对1个单一变量计算一样），可以更快速的实现梯度计算。计算梯度的代码中直接用$(z_1 - y_1) \\cdot x_1$，得到的是一个13维的向量，每个分量分别代表该维度的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample1 [  0.          23.48051799   9.58029163   0.          41.06674958\n",
      "  75.33401592  83.69586171  35.11682862   0.          29.68425495\n",
      "  37.46891169 130.44732219  11.69850434], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = (z1 - y1) * x1\n",
    "print('gradient_w_by_sample1 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "输入数据中有多个样本，每个样本都对梯度有贡献。如上代码计算了只有样本1时的梯度值，同样的计算方法也可以计算样本2和样本3对梯度的贡献。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample2 [2.54738434e-02 0.00000000e+00 2.83333765e+01 0.00000000e+00\n",
      " 1.86624242e+01 5.91703008e+01 8.45121992e+01 3.76793284e+01\n",
      " 4.69458498e+00 1.23980167e+01 5.97311025e+01 1.07975454e+02\n",
      " 2.20777626e+01], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "x2 = x[1]\n",
    "y2 = y[1]\n",
    "z2 = net.forward(x2)\n",
    "gradient_w = (z2 - y2) * x2\n",
    "print('gradient_w_by_sample2 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample3 [3.07963708e-02 0.00000000e+00 3.42860463e+01 0.00000000e+00\n",
      " 2.25832858e+01 9.07287666e+01 7.83155260e+01 4.55955257e+01\n",
      " 5.68088867e+00 1.50027645e+01 7.22802431e+01 1.29029688e+02\n",
      " 8.29246719e+00], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "x3 = x[2]\n",
    "y3 = y[2]\n",
    "z3 = net.forward(x3)\n",
    "gradient_w = (z3 - y3) * x3\n",
    "print('gradient_w_by_sample3 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可能有的读者再次想到可以使用for循环把每个样本对梯度的贡献都计算出来，然后再作平均。但是我们不需要这么做，仍然可以使用Numpy的矩阵操作来简化运算，如3个样本的情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x [[0.00000000e+00 1.80000000e-01 7.34418420e-02 0.00000000e+00\n",
      "  3.14814815e-01 5.77505269e-01 6.41606591e-01 2.69203139e-01\n",
      "  0.00000000e+00 2.27557411e-01 2.87234043e-01 1.00000000e+00\n",
      "  8.96799117e-02]\n",
      " [2.35922539e-04 0.00000000e+00 2.62405717e-01 0.00000000e+00\n",
      "  1.72839506e-01 5.47997701e-01 7.82698249e-01 3.48961980e-01\n",
      "  4.34782609e-02 1.14822547e-01 5.53191489e-01 1.00000000e+00\n",
      "  2.04470199e-01]\n",
      " [2.35697744e-04 0.00000000e+00 2.62405717e-01 0.00000000e+00\n",
      "  1.72839506e-01 6.94385898e-01 5.99382080e-01 3.48961980e-01\n",
      "  4.34782609e-02 1.14822547e-01 5.53191489e-01 9.87519166e-01\n",
      "  6.34657837e-02]], shape (3, 13)\n",
      "y [[0.42222222]\n",
      " [0.36888889]\n",
      " [0.66      ]], shape (3, 1)\n",
      "z [[130.86954441]\n",
      " [108.34434338]\n",
      " [131.3204395 ]], shape (3, 1)\n"
     ]
    }
   ],
   "source": [
    "# 注意这里是一次取出3个样本的数据，不是取出第3个样本\n",
    "x3samples = x[0:3]\n",
    "y3samples = y[0:3]\n",
    "z3samples = net.forward(x3samples)\n",
    "\n",
    "print('x {}, shape {}'.format(x3samples, x3samples.shape))\n",
    "print('y {}, shape {}'.format(y3samples, y3samples.shape))\n",
    "print('z {}, shape {}'.format(z3samples, z3samples.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面的x3samples, y3samples, z3samples的第一维大小均为3，表示有3个样本。下面计算这3个样本对梯度的贡献。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w [[0.00000000e+00 2.34805180e+01 9.58029163e+00 0.00000000e+00\n",
      "  4.10667496e+01 7.53340159e+01 8.36958617e+01 3.51168286e+01\n",
      "  0.00000000e+00 2.96842549e+01 3.74689117e+01 1.30447322e+02\n",
      "  1.16985043e+01]\n",
      " [2.54738434e-02 0.00000000e+00 2.83333765e+01 0.00000000e+00\n",
      "  1.86624242e+01 5.91703008e+01 8.45121992e+01 3.76793284e+01\n",
      "  4.69458498e+00 1.23980167e+01 5.97311025e+01 1.07975454e+02\n",
      "  2.20777626e+01]\n",
      " [3.07963708e-02 0.00000000e+00 3.42860463e+01 0.00000000e+00\n",
      "  2.25832858e+01 9.07287666e+01 7.83155260e+01 4.55955257e+01\n",
      "  5.68088867e+00 1.50027645e+01 7.22802431e+01 1.29029688e+02\n",
      "  8.29246719e+00]], gradient.shape (3, 13)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = (z3samples - y3samples) * x3samples\n",
    "print('gradient_w {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处可见，计算梯度`gradient_w`的维度是$3 \\times 13$，并且其第1行与上面第1个样本计算的梯度gradient_w_by_sample1一致，第2行与上面第2个样本计算的梯度gradient_w_by_sample2一致，第3行与上面第3个样本计算的梯度gradient_w_by_sample3一致。这里使用矩阵操作，可以更加方便的对3个样本分别计算各自对梯度的贡献。\n",
    "\n",
    "那么对于有N个样本的情形，我们可以直接使用如下方式计算出所有样本对梯度的贡献，这就是使用Numpy库广播功能带来的便捷。\n",
    "小结一下这里使用Numpy库的广播功能：\n",
    "- 一方面可以扩展参数的维度，代替for循环来计算1个样本对从$w_0$到$w_12$的所有参数的梯度。\n",
    "- 另一方面可以扩展样本的维度，代替for循环来计算样本0到样本403对参数的梯度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w shape (404, 13)\n",
      "[[0.00000000e+00 2.34805180e+01 9.58029163e+00 ... 3.74689117e+01\n",
      "  1.30447322e+02 1.16985043e+01]\n",
      " [2.54738434e-02 0.00000000e+00 2.83333765e+01 ... 5.97311025e+01\n",
      "  1.07975454e+02 2.20777626e+01]\n",
      " [3.07963708e-02 0.00000000e+00 3.42860463e+01 ... 7.22802431e+01\n",
      "  1.29029688e+02 8.29246719e+00]\n",
      " ...\n",
      " [3.97706874e+01 0.00000000e+00 1.74130673e+02 ... 2.01043762e+02\n",
      "  2.48659390e+02 1.27554582e+02]\n",
      " [2.69696515e+01 0.00000000e+00 1.75225687e+02 ... 2.02308019e+02\n",
      "  2.34270491e+02 1.28287658e+02]\n",
      " [6.08972123e+01 0.00000000e+00 1.53017134e+02 ... 1.76666981e+02\n",
      "  2.18509161e+02 1.08772220e+02]]\n"
     ]
    }
   ],
   "source": [
    "z = net.forward(x)\n",
    "gradient_w = (z - y) * x\n",
    "print('gradient_w shape {}'.format(gradient_w.shape))\n",
    "print(gradient_w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面gradient_w的每一行代表了一个样本对梯度的贡献。根据梯度的计算公式，总梯度是对每个样本对梯度贡献的平均值。\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)\\frac{\\partial{z_i}}{\\partial{w_j}}} = \\frac{1}{N}\\sum_{i=1}^N{(z_i - y_i)x_i^{j}}$$\n",
    "\n",
    "我们也可以使用Numpy的均值函数来完成此过程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w  (13,)\n",
      "w  (13, 1)\n",
      "[  4.6555403   19.35268996  55.88081118  14.00266972  47.98588869\n",
      "  76.87210821  94.8555119   36.07579608  45.44575958  59.65733292\n",
      "  83.65114918 134.80387478  38.93998153]\n",
      "[[ 1.76405235e+00]\n",
      " [ 4.00157208e-01]\n",
      " [ 9.78737984e-01]\n",
      " [ 2.24089320e+00]\n",
      " [ 1.86755799e+00]\n",
      " [ 1.59000000e+02]\n",
      " [ 9.50088418e-01]\n",
      " [-1.51357208e-01]\n",
      " [-1.03218852e-01]\n",
      " [ 1.59000000e+02]\n",
      " [ 1.44043571e-01]\n",
      " [ 1.45427351e+00]\n",
      " [ 7.61037725e-01]]\n"
     ]
    }
   ],
   "source": [
    "# axis = 0 表示把每一行做相加然后再除以总的行数\n",
    "gradient_w = np.mean(gradient_w, axis=0)\n",
    "print('gradient_w ', gradient_w.shape)\n",
    "print('w ', net.w.shape)\n",
    "print(gradient_w)\n",
    "print(net.w)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用Numpy的矩阵操作方便地完成了gradient的计算，但引入了一个问题，`gradient_w`的形状是(13,)，而$w$的维度是(13, 1)。导致该问题的原因是使用`np.mean`函数时消除了第0维。为了加减乘除等计算方便，`gradient_w`和$w$必须保持一致的形状。因此我们将`gradient_w`的维度也设置为(13,1)，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w shape (13, 1)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = gradient_w[:, np.newaxis]\n",
    "print('gradient_w shape', gradient_w.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "综合上面的剖析，计算梯度的代码如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  4.6555403 ],\n",
       "       [ 19.35268996],\n",
       "       [ 55.88081118],\n",
       "       [ 14.00266972],\n",
       "       [ 47.98588869],\n",
       "       [ 76.87210821],\n",
       "       [ 94.8555119 ],\n",
       "       [ 36.07579608],\n",
       "       [ 45.44575958],\n",
       "       [ 59.65733292],\n",
       "       [ 83.65114918],\n",
       "       [134.80387478],\n",
       "       [ 38.93998153]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = net.forward(x)\n",
    "gradient_w = (z - y) * x\n",
    "gradient_w = np.mean(gradient_w, axis=0)\n",
    "gradient_w = gradient_w[:, np.newaxis]\n",
    "gradient_w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上述代码非常简洁地完成了$w$的梯度计算。同样，计算$b$的梯度的代码也是类似的原理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "142.50289323156107"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient_b = (z - y)\n",
    "gradient_b = np.mean(gradient_b)\n",
    "# 此处b是一个数值，所以可以直接用np.mean得到一个标量\n",
    "gradient_b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "将上面计算$w$和$b$的梯度的过程，写成Network类的`gradient`函数，实现方法如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)\n",
    "        \n",
    "        return gradient_w, gradient_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "point [-100.0, -100.0], loss 7873.345739941161\n",
      "gradient [-45.87968288123223, -35.50236884482904]\n"
     ]
    }
   ],
   "source": [
    "# 调用上面定义的gradient函数，计算梯度\n",
    "# 初始化网络\n",
    "net = Network(13)\n",
    "# 设置[w5, w9] = [-100., -100.]\n",
    "net.w[5] = -100.0\n",
    "net.w[9] = -100.0\n",
    "\n",
    "z = net.forward(x)\n",
    "loss = net.loss(z, y)\n",
    "gradient_w, gradient_b = net.gradient(x, y)\n",
    "gradient_w5 = gradient_w[5][0]\n",
    "gradient_w9 = gradient_w[9][0]\n",
    "print('point {}, loss {}'.format([net.w[5][0], net.w[9][0]], loss))\n",
    "print('gradient {}'.format([gradient_w5, gradient_w9]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 确定损失函数更小的点\n",
    "\n",
    "下面我们开始研究更新梯度的方法。首先沿着梯度的反方向移动一小步，找到下一个点P1，观察损失函数的变化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "point [-95.41203171187678, -96.4497631155171], loss 7214.694816482369\n",
      "gradient [-43.883932999069096, -34.019273908495926]\n"
     ]
    }
   ],
   "source": [
    "# 在[w5, w9]平面上，沿着梯度的反方向移动到下一个点P1\n",
    "# 定义移动步长 eta\n",
    "eta = 0.1\n",
    "# 更新参数w5和w9\n",
    "net.w[5] = net.w[5] - eta * gradient_w5\n",
    "net.w[9] = net.w[9] - eta * gradient_w9\n",
    "# 重新计算z和loss\n",
    "z = net.forward(x)\n",
    "loss = net.loss(z, y)\n",
    "gradient_w, gradient_b = net.gradient(x, y)\n",
    "gradient_w5 = gradient_w[5][0]\n",
    "gradient_w9 = gradient_w[9][0]\n",
    "print('point {}, loss {}'.format([net.w[5][0], net.w[9][0]], loss))\n",
    "print('gradient {}'.format([gradient_w5, gradient_w9]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "运行上面的代码，可以发现沿着梯度反方向走一小步，下一个点的损失函数的确减少了。感兴趣的话，大家可以尝试不停的点击上面的代码块，观察损失函数是否一直在变小。\n",
    "\n",
    "在上述代码中，每次更新参数使用的语句：\n",
    "`net.w[5] = net.w[5] - eta * gradient_w5`\n",
    "\n",
    "* 相减：参数需要向梯度的反方向移动。\n",
    "* eta：控制每次参数值沿着梯度反方向变动的大小，即每次移动的步长，又称为学习率。\n",
    "\n",
    "大家可以思考下，为什么之前我们要做输入特征的归一化，保持尺度一致？这是为了让统一的步长更加合适。\n",
    "\n",
    "如 **图8** 所示，特征输入归一化后，不同参数输出的Loss是一个比较规整的曲线，学习率可以设置成统一的值 ；特征输入未归一化时，不同特征对应的参数所需的步长不一致，尺度较大的参数需要大步长，尺寸较小的参数需要小步长，导致无法设置统一的学习率。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/903f552bc55b4a5eba71caa7dd86fd2d7b71b8ebb6cb4500a5f5711f465707f3\" width=\"300\" hegiht=\"40\" ></center>\n",
    "<center><br>图8：未归一化的特征，会导致不同特征维度的理想步长不同</br></center>\n",
    "<br></br>\n",
    "\n",
    "###  代码封装Train函数\n",
    "\n",
    "将上面的循环计算过程封装在`train`和`update`函数中，实现方法如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter 0, point [-99.54120317118768, -99.64497631155172], loss 7873.345739941161\n",
      "iter 50, point [-78.9761810944732, -83.65939206734069], loss 5131.480704109405\n",
      "iter 100, point [-62.4493631356931, -70.67918223434114], loss 3346.754494352463\n",
      "iter 150, point [-49.17799206644332, -60.12620415441553], loss 2184.906016270654\n",
      "iter 200, point [-38.53070194231174, -51.533984751788346], loss 1428.4172504483342\n",
      "iter 250, point [-29.998249130283174, -44.52613603923428], loss 935.7392894242679\n",
      "iter 300, point [-23.169901624519575, -38.79894318028118], loss 614.7592258739251\n",
      "iter 350, point [-17.71439280083778, -34.10731848231335], loss 405.53408184471505\n",
      "iter 400, point [-13.364557220746388, -30.253470630210863], loss 269.0551396220099\n",
      "iter 450, point [-9.904936677384967, -27.077764259976597], loss 179.9364750604248\n",
      "iter 500, point [-7.161782280775628, -24.451346444229817], loss 121.65711285489998\n",
      "iter 550, point [-4.994989383373879, -22.270198517465555], loss 83.46491706360901\n",
      "iter 600, point [-3.2915916915280783, -20.450337700789422], loss 58.36183370758033\n",
      "iter 650, point [-1.9605131425212885, -18.923946252536773], loss 41.792808952534\n",
      "iter 700, point [-0.9283343968114077, -17.636248840494844], loss 30.792614998570482\n",
      "iter 750, point [-0.13587780041668718, -16.542993494033716], loss 23.43065354742935\n",
      "iter 800, point [0.4645474092373408, -15.60841945615185], loss 18.449664464381506\n",
      "iter 850, point [0.9113672926170796, -14.803617811655524], loss 15.030615923519784\n",
      "iter 900, point [1.2355357562745004, -14.105208963393421], loss 12.639705730905764\n",
      "iter 950, point [1.4619805189121953, -13.494275706622066], loss 10.928795653764196\n",
      "iter 1000, point [1.6107694974712377, -12.955502492189021], loss 9.670616807081698\n",
      "iter 1050, point [1.6980516626374353, -12.476481020835202], loss 8.716602071285436\n",
      "iter 1100, point [1.7368159644039771, -12.04715001603925], loss 7.969442965176621\n",
      "iter 1150, point [1.7375034995020395, -11.659343238414994], loss 7.365228465612388\n",
      "iter 1200, point [1.7085012931271857, -11.306424818680442], loss 6.861819342703047\n",
      "iter 1250, point [1.6565405824483015, -10.982995030930885], loss 6.431280353078019\n",
      "iter 1300, point [1.5870180647823104, -10.684652890749808], loss 6.054953198278096\n",
      "iter 1350, point [1.5042550040699705, -10.407804594738165], loss 5.720248083137862\n",
      "iter 1400, point [1.4117062100403601, -10.14950894127009], loss 5.418553777303124\n",
      "iter 1450, point [1.3121285818148223, -9.907352585055445], loss 5.143875665274019\n",
      "iter 1500, point [1.2077170340724794, -9.67934935975478], loss 4.891947653805328\n",
      "iter 1550, point [1.1002141124777076, -9.463859017459276], loss 4.659652555766873\n",
      "iter 1600, point [0.990998385834045, -9.259521632951046], loss 4.444643323159747\n",
      "iter 1650, point [0.8811557188942747, -9.065204645952335], loss 4.245095084874306\n",
      "iter 1700, point [0.7715367363576023, -8.87996009965401], loss 4.059542401818773\n",
      "iter 1750, point [0.662803148565214, -8.702990105791185], loss 3.88677206759292\n",
      "iter 1800, point [0.5554650931141796, -8.533618947271485], loss 3.7257521401326525\n",
      "iter 1850, point [0.4499112301277286, -8.371270536496699], loss 3.5755846299900256\n",
      "iter 1900, point [0.3464329929523944, -8.215450195281456], loss 3.4354736574042524\n",
      "iter 1950, point [0.24524412503452966, -8.065729922139326], loss 3.3047037451160453\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3X10XPV95/H3V8+yZD3ZsixLBtlgICaEJwVMnjYJiTEkjWnzcMjmFJd6121Dd5Nmuy3ZdJdu0uxJ2m2S0gd6nODE9CQhhJLizbIhjiFtEoJBBmPAYCSMjS1sS7b8bMt6+u4f85MZC401Y83MleZ+XufMmXt/85s737mS5qN77+/ONXdHRETipyjqAkREJBoKABGRmFIAiIjElAJARCSmFAAiIjGlABARiSkFgIhITCkARERiSgEgIhJTJVEXcDazZ8/2tra2qMsQEZlWNm3atN/dGyfqN6UDoK2tjY6OjqjLEBGZVsxsZzr9tAtIRCSm0goAM/sjM3vBzJ43s++bWYWZLTCzjWbWZWY/MLOy0Lc8zHeFx9uSlvP50L7NzG7IzVsSEZF0TBgAZtYC/Geg3d3fChQDtwBfBb7u7hcCB4GV4SkrgYOh/euhH2a2ODzvUmAZ8A9mVpzdtyMiIulKdxdQCVBpZiXADGAP8H7ggfD4WuDmML08zBMev97MLLTf5+6n3P1VoAu4ZvJvQUREzsWEAeDu3cD/Bl4j8cF/GNgEHHL3odBtN9ASpluAXeG5Q6H/rOT2cZ4jIiJ5ls4uoHoS/70vAOYBVSR24eSEma0ysw4z6+jt7c3Vy4iIxF46u4A+ALzq7r3uPgg8CLwTqAu7hABage4w3Q3MBwiP1wIHktvHec5p7r7a3dvdvb2xccJhrCIico7SCYDXgCVmNiPsy78e2Ao8Bnws9FkBPBSm14V5wuOPeuK6k+uAW8IooQXAIuDJ7LyNM3UfOsnXfrqNHfuP52LxIiIFIZ1jABtJHMx9GnguPGc18KfA58ysi8Q+/nvCU+4BZoX2zwF3hOW8ANxPIjx+Atzu7sNZfTfBoRMD3PVoFy/tPZKLxYuIFIS0zgR29zuBO8c0b2ecUTzu3g98PMVyvgx8OcMaM9Y4sxyA3qOncv1SIiLTVkGeCTyrqpwiUwCIiJxNQQZAcZExq7qcHgWAiEhKBRkAAI3V5doCEBE5i4INgDk15fQeUwCIiKRSsAHQWF1OzxEFgIhIKoUbADPL2X/sFCMjHnUpIiJTUsEGwJyZ5QyNOIdODkZdiojIlFSwAdA4swKAnqP9EVciIjI1FXAA6GQwEZGzKdgAmKMAEBE5q4INgNEtAJ0MJiIyvoINgKryEmaUFWsLQEQkhYINAEhsBSgARETGV9ABMGdmuUYBiYikUNABoC0AEZHUCjsA9IVwIiIpFXQAzKmp4Ej/EP2DObnwmIjItDZhAJjZxWa2Oel2xMw+a2YNZrbezDrDfX3ob2Z2l5l1mdkWM7sqaVkrQv9OM1uR+lWzo7Fa5wKIiKSSzjWBt7n7Fe5+BXA1cAL4EYlr/W5w90XAhjAPcCOJC74vAlYBdwOYWQOJy0peS+JSkneOhkaunD4bWF8LLSLyJpnuAroeeMXddwLLgbWhfS1wc5heDtzrCU8AdWbWDNwArHf3Pnc/CKwHlk36HZzF6ZPBjmgkkIjIWJkGwC3A98N0k7vvCdN7gaYw3QLsSnrO7tCWqv0MZrbKzDrMrKO3tzfD8s7UXJv4Qri9hxUAIiJjpR0AZlYGfAT44djH3N2BrHzxvruvdvd2d29vbGyc1LIaqsooKy5ij7YARETeJJMtgBuBp919X5jfF3btEO57Qns3MD/pea2hLVV7zpgZc2rK2actABGRN8kkAD7JG7t/ANYBoyN5VgAPJbXfGkYDLQEOh11FjwBLzaw+HPxdGtpyam5NBXu1BSAi8iYl6XQysyrgg8DvJTV/BbjfzFYCO4FPhPaHgZuALhIjhm4DcPc+M/sS8FTo90V375v0O5hAU20FW18/kuuXERGZdtIKAHc/Dswa03aAxKigsX0duD3FctYAazIv89w111Tw6Is9uDtmls+XFhGZ0gr6TGCAubUVnBwc5kj/UNSliIhMKQUfAE01GgoqIjKegg+AuaPnAuhAsIjIGQo/AMIWgIaCioicqeADYE5N4usgtAUgInKmgg+A8pJiZlWVKQBERMYo+ACAxIFg7QISETlTLAJgbm0FexQAIiJniEUANNVUsE+7gEREzhCLAJhbU8GB4wOcGtKlIUVERsUiAEavC9BzRFcGExEZFYsAaNLJYCIibxKLAJgXAuD1QycjrkREZOqIRwDUVQLw+iFtAYiIjIpFAFSVl1A3o5TuQyeiLkVEZMqIRQAAtNRVagtARCRJWgFgZnVm9oCZvWRmL5rZdWbWYGbrzawz3NeHvmZmd5lZl5ltMbOrkpazIvTvNLMVqV8x++bVVdJ9UMcARERGpbsF8DfAT9z9EuBy4EXgDmCDuy8CNoR5SFw8flG4rQLuBjCzBuBO4FrgGuDO0dDIh8QWgAJARGTUhAFgZrXAe4B7ANx9wN0PAcuBtaHbWuDmML0cuNcTngDqzKwZuAFY7+597n4QWA8sy+q7OYuWukqOnhri8MnBfL2kiMiUls4WwAKgF/i2mT1jZt8KF4lvcvc9oc9eoClMtwC7kp6/O7Slaj+Dma0ysw4z6+jt7c3s3ZzFGyOBtBUgIgLpBUAJcBVwt7tfCRznjd09wOkLwXs2CnL31e7e7u7tjY2N2VgkAC31CgARkWTpBMBuYLe7bwzzD5AIhH1h1w7hvic83g3MT3p+a2hL1Z4X8+oSJ4N1KwBERIA0AsDd9wK7zOzi0HQ9sBVYB4yO5FkBPBSm1wG3htFAS4DDYVfRI8BSM6sPB3+Xhra8mF1VTllxkQJARCQoSbPffwK+a2ZlwHbgNhLhcb+ZrQR2Ap8IfR8GbgK6gBOhL+7eZ2ZfAp4K/b7o7n1ZeRdpKCoy5tVVaCioiEiQVgC4+2agfZyHrh+nrwO3p1jOGmBNJgVm0zwNBRUROS02ZwJDYiiodgGJiCTEKgDm1VXSc/QUA0MjUZciIhK5WAVAS10l7ujykCIixC0AwrkAuw7qW0FFRGIVAOc1zABgd5+OA4iIxCoAmmsrKC4yXuvTFoCISKwCoKS4iJa6SgWAiAgxCwBI7AZSAIiIxDAA5jfMYJcCQEQkfgFwXsMMDhwf4NipoahLERGJVCwDANBWgIjEXmwDQMcBRCTuYhsA2gIQkbiLXQDUziilpqJEWwAiEnuxCwCA82ZpKKiISDwDQOcCiIikFwBmtsPMnjOzzWbWEdoazGy9mXWG+/rQbmZ2l5l1mdkWM7sqaTkrQv9OM1uR6vVybX7DDHb3nWRkJCvXsRcRmZYy2QJ4n7tf4e6jVwa7A9jg7ouADWEe4EZgUbitAu6GRGAAdwLXAtcAd46GRr6d1zCDgeER9h3V10KLSHxNZhfQcmBtmF4L3JzUfq8nPAHUmVkzcAOw3t373P0gsB5YNonXP2ejI4F27NduIBGJr3QDwIGfmtkmM1sV2prcfU+Y3gs0hekWYFfSc3eHtlTtZzCzVWbWYWYdvb29aZaXmQWzqwDYceB4TpYvIjIdpHVReOBd7t5tZnOA9Wb2UvKD7u5mlpUd6u6+GlgN0N7enpOd9PNqKykrKeLV/QoAEYmvtLYA3L073PcAPyKxD39f2LVDuO8J3buB+UlPbw1tqdrzrqjIWDCriu29CgARia8JA8DMqsxs5ug0sBR4HlgHjI7kWQE8FKbXAbeG0UBLgMNhV9EjwFIzqw8Hf5eGtkgsmF3Fq/uPRfXyIiKRS2cXUBPwIzMb7f89d/+JmT0F3G9mK4GdwCdC/4eBm4Au4ARwG4C795nZl4CnQr8vuntf1t5JhhY0VrHhpX0MDY9QUhzL0yFEJOYmDAB33w5cPk77AeD6cdoduD3FstYAazIvM/sWzK5icNjpPnSS82dVRV2OiEjexfZf34VhJNB2HQgWkZiKbQCMDgV9VQeCRSSmYhsADVVl1FSUaCioiMRWbAPAzFjQWK0AEJHYim0AQOI4gAJAROIq1gGwYHYV3YdO0j84HHUpIiJ5F/sAALQVICKxFOsAuHBONQCdPTojWETiJ9YBsLCxiiKDrn1Hoy5FRCTvYh0A5SXFtM2q4uV92gIQkfiJdQAALGqqprNHWwAiEj8KgDkz2XHgBKeGNBJIROJFAdBUzfCI6/KQIhI7CoA5MwF4WQeCRSRmYh8AoyOBNBRUROIm9gFQUVrM+bOq6NKBYBGJmbQDwMyKzewZM/txmF9gZhvNrMvMfmBmZaG9PMx3hcfbkpbx+dC+zcxuyPabOVeL5lRrKKiIxE4mWwCfAV5Mmv8q8HV3vxA4CKwM7SuBg6H966EfZrYYuAW4FFgG/IOZFU+u/OxY1FTNjv3HGRgaiboUEZG8SSsAzKwV+BDwrTBvwPuBB0KXtcDNYXp5mCc8fn3ovxy4z91PufurJK4ZfE023sRkXdQ0k6ER13cCiUispLsF8A3gT4DRf5FnAYfcfSjM7wZawnQLsAsgPH449D/dPs5zInXJ3BoAXtxzJOJKRETyZ8IAMLMPAz3uvikP9WBmq8ysw8w6ent78/GSLGysoqykSAEgIrGSzhbAO4GPmNkO4D4Su37+Bqgzs5LQpxXoDtPdwHyA8HgtcCC5fZznnObuq9293d3bGxsbM35D56K0uIiLmqrZqgAQkRiZMADc/fPu3urubSQO4j7q7p8CHgM+FrqtAB4K0+vCPOHxR93dQ/stYZTQAmAR8GTW3skkLW6uYevrR0iUKiJS+CZzHsCfAp8zsy4S+/jvCe33ALNC++eAOwDc/QXgfmAr8BPgdnefMl/A85bmGg4cH6D36KmoSxERyYuSibu8wd1/Dvw8TG9nnFE87t4PfDzF878MfDnTIvNhcXPiQPDWPUeYU1MRcTUiIrkX+zOBR71l3hsBICISBwqAoKailNb6Sl7co6+EEJF4UAAkSRwIPhx1GSIieaEASPKW5hpe3X+ckwNT5ti0iEjOKACSXDqvhhHXcQARiQcFQJK3tdYBsGX3oYgrERHJPQVAkrm1FcyZWc6W3ToOICKFTwEwxtta63hWWwAiEgMKgDEub61le+9xjvQPRl2KiEhOKQDGeNv8xHGA57UbSEQKnAJgjMtbawF4VgEgIgVOATBG3Ywyzp81QyOBRKTgKQDG8bbWOp7dpQAQkcKmABjH5a21vH64n56j/VGXIiKSMwqAcVx5Xj0AT+/UVoCIFC4FwDje2lJDWUkRHTv6oi5FRCRn0rkofIWZPWlmz5rZC2b2P0P7AjPbaGZdZvYDMysL7eVhvis83pa0rM+H9m1mdkOu3tRklZcUc3lrLR07D0ZdiohIzqSzBXAKeL+7Xw5cASwzsyXAV4Gvu/uFwEFgZei/EjgY2r8e+mFmi0lcU/hSYBnwD2ZWnM03k03tbQ08331Y3wwqIgUrnYvCu7sfC7Ol4ebA+4EHQvta4OYwvTzMEx6/3swstN/n7qfc/VWgi3EuKTlVtJ9fz9CI62shRKRgpXUMwMyKzWwz0AOsB14BDrn7UOiyG2gJ0y3ALoDw+GESF40/3T7Oc6acq89PHAjWcQARKVRpBYC7D7v7FUArif/aL8lVQWa2ysw6zKyjt7c3Vy8zoboZZSyaU63jACJSsDIaBeTuh4DHgOuAOjMrCQ+1At1huhuYDxAerwUOJLeP85zk11jt7u3u3t7Y2JhJeVnX3tbApp0HGRnxSOsQEcmFdEYBNZpZXZiuBD4IvEgiCD4Wuq0AHgrT68I84fFH3d1D+y1hlNACYBHwZLbeSC68va2eo/1DvLhXVwgTkcJTMnEXmoG1YcROEXC/u//YzLYC95nZXwDPAPeE/vcA/2RmXUAfiZE/uPsLZnY/sBUYAm539yk9xOa6C2YB8OtXDnDpvNqIqxERya4JA8DdtwBXjtO+nXFG8bh7P/DxFMv6MvDlzMuMRnNtJQtnV/H4Kwf4D+9eGHU5IiJZpTOBJ/COC2excfsBBodHoi5FRCSrFAATeMcFszk+MKzrBItIwVEATGDJwtHjAPsjrkREJLsUABNoqCpjcXMNv+o6EHUpIiJZpQBIwzsumMWm1w7SPzilBy2JiGREAZCGd1/UyMDQCL/erq0AESkcCoA0XLuggYrSIn7+Uk/UpYiIZI0CIA0VpcW884LZPLqth8RJzSIi058CIE3vu2QOu/pO8krv8ahLERHJCgVAmt53yRwAHtNuIBEpEAqANLXUVXJx00we26YAEJHCoADIwHsvaeTJV/s42j8YdSkiIpOmAMjA9Zc0MTTiPLYtugvViIhkiwIgA1efX0/jzHJ+8vyeqEsREZk0BUAGiouMZZfO5bGXejkxMDTxE0REpjAFQIZuvGwuJweH+VftBhKRaS6dS0LON7PHzGyrmb1gZp8J7Q1mtt7MOsN9fWg3M7vLzLrMbIuZXZW0rBWhf6eZrUj1mlPZNW0NzKoq4/8+p91AIjK9pbMFMAT8F3dfDCwBbjezxcAdwAZ3XwRsCPMAN5K43u8iYBVwNyQCA7gTuJbElcTuHA2N6aSkuIill87l0Zd69OVwIjKtTRgA7r7H3Z8O00dJXBC+BVgOrA3d1gI3h+nlwL2e8ARQZ2bNwA3Aenfvc/eDwHpgWVbfTZ7cdNlcTgwM83OdEyAi01hGxwDMrI3E9YE3Ak3uProfZC/QFKZbgF1JT9sd2lK1TzvXLZzF7OpyHny6O+pSRETOWdoBYGbVwD8Dn3X3I8mPeeIb0rLyLWlmtsrMOsyso7d3ah5oLSku4uYr5vHYth76jg9EXY6IyDlJKwDMrJTEh/933f3B0Lwv7Noh3I/uD+kG5ic9vTW0pWo/g7uvdvd2d29vbGzM5L3k1UevbmVw2Fm3WVsBIjI9pTMKyIB7gBfd/WtJD60DRkfyrAAeSmq/NYwGWgIcDruKHgGWmll9OPi7NLRNS29prmFxcw0PPqMAEJHpKZ0tgHcCvw2838w2h9tNwFeAD5pZJ/CBMA/wMLAd6AK+CXwawN37gC8BT4XbF0PbtPXRq1vZsvswnfuORl2KiEjGbCpf4KS9vd07OjqiLiOl/cdOseR/beB33tHGn314cdTliIgAYGab3L19on46E3gSZleXc8Nb5/LDTbs5OaBzAkRkelEATNKtS87n8MlB/s+zr0ddiohIRhQAk3TNggYuaqrm3id26HrBIjKtKAAmycz47evaeL77CM/sOhR1OSIiaVMAZMFvXtlCdXkJ3/nVjqhLERFJmwIgC6rLS/j3157Hj7e8zmsHTkRdjohIWhQAWbLyXQsoKSpi9S9eiboUEZG0KACypKmmgo9e3cL9HbvpOdofdTkiIhNSAGTR773nAoaGR1jzyx1RlyIiMiEFQBa1za7iw2+bx9rHd9B79FTU5YiInJUCIMs+98GLGBwe4e8e7Yy6FBGRs1IAZFnb7Co+8fb5fO/J1zQiSESmNAVADnzm+kUUmfG19duiLkVEJCUFQA401VSw8l0L+JfNr7Np57T+xmsRKWAKgBy5/X0X0lxbwX//lxcYHtF3BInI1KMAyJGq8hK+8KG3sHXPEb67cWfU5YiIvEk6l4RcY2Y9ZvZ8UluDma03s85wXx/azczuMrMuM9tiZlclPWdF6N9pZivGe61C86HLmnnnhbP4q0e2sfewTg4TkaklnS2A7wDLxrTdAWxw90XAhjAPcCOwKNxWAXdDIjCAO4FrgWuAO0dDo5CZGX9x82UMDo9wx4Nb9HXRIjKlTBgA7v5vwNgjmcuBtWF6LXBzUvu9nvAEUGdmzcANwHp373P3g8B63hwqBWnB7CruWHYJP9/Wy/0du6IuR0TktHM9BtDk7nvC9F6gKUy3AMmfcrtDW6r2WLj1ujauWziLL/34RXb16dwAEZkaJn0Q2BP7NbK2b8PMVplZh5l19Pb2ZmuxkSoqMv7yY2/DDD793afpH9T1g0UkeucaAPvCrh3CfU9o7wbmJ/VrDW2p2t/E3Ve7e7u7tzc2Np5jeVPP/IYZ/PXHL+e57sN88cdboy5HROScA2AdMDqSZwXwUFL7rWE00BLgcNhV9Aiw1Mzqw8HfpaEtVpZeOpff/3cX8L2Nr+l4gIhErmSiDmb2feC9wGwz201iNM9XgPvNbCWwE/hE6P4wcBPQBZwAbgNw9z4z+xLwVOj3RXeP5Smyf7z0Ip7rPsR/e/A5WuoqeeeFs6MuSURiyqby0MT29nbv6OiIuoysO9I/yMfv/jWvHzrJD//gOi6ZWxN1SSJSQMxsk7u3T9RPZwJHoKailG/f9nZmlBezYs2TbO89FnVJIhJDCoCIzKur5N7fvZahYeeT33xCISAieacAiNDFc2fyvf+4hKFh55bVT/DyvqNRlyQiMaIAiNhoCDjw0bsf5/FX9kddkojEhAJgCrh47kx+9Ol3MLemghVrnuSHGiIqInmgAJgiWutn8MAfvIO3tzXwXx/Ywh3/vEVnDItITikAppDaylLu/d1r+PR7L+C+p3Zx89//im17dVxARHJDATDFlBQX8SfLLuHbt72dnqOn+PDf/oJv/OxlBoZGoi5NRAqMAmCKet/Fc1j/R+/hxrc2842fdfIbf/tLftFZGF+OJyJTgwJgCptVXc5dn7ySb93azvGBIX77nif5nW8/qd1CIpIV+iqIaeLU0DD3Pr6Tv320kyP9Qyxd3MSn33chV8yvi7o0EZli0v0qCAXANHPw+ADf/tWrfOfxHRzpH2LJwgY+de35LL20ifKS4qjLE5EpQAFQ4I6dGuJ7G3ey9vGddB86Sf2MUn7rqlZ+4/J5XN5ai5lFXaKIREQBEBPDI84vu/Zz35OvsX7rPoZGnJa6Sm5861w+sLiJq86rp6xEh3pE4kQBEEOHTgywfus+fvL8Xn7RuZ+B4REqS4u5dmED77pwNm9va+AtzTUKBJECpwCIuaP9g/z6lQP8sms/v+zaz/be4wCUlRRx6bwaLm+t47KWWhY1VXPhnGpmlE14bSARmSambACY2TLgb4Bi4Fvu/pVUfRUA2bPn8Emeee0Qm3clbs/tPszJpK+aaK2vZNGcatpmV9FaP4PW+spwm0FtZWmElYtIptINgLz+22dmxcDfAx8EdgNPmdk6d9dV0nOsubaS5ssquemyZgCGhkfY2XeCzn3H6Nx3lM6eY7y87ygbX+3jxMCZ30E0s7yExpnlzKouY3Z1ObOr35huqCpjZkUJMytKqQn3MytKqCjViCSRqS7f2/3XAF3uvh3AzO4DlgMKgDwrKS7igsZqLmisZtlb555ud3cOnRhk98GT7D54gt0HT9J96CS9x05x4NgpOnuO8evtBzh0YvCsyy8rKaKmooSq8hIqS4spLy2msrSIitJiKkqKqSwrpmJ0PrSVFBulxUZJUVHivriIkiKjtLiIkjHtpUWJ++Iio7jIKDIoMsOMMJ9oM0tMF4fHipL6jvYZnbYiKB6dDoOoTt9jp+dHx1eZWdI0Gnkl006+A6AFSP6u493AtXmuQc7CzKivKqO+qozLWmtT9hscHqHv+AB9xwc42j/E0f5BjvYPcSTp/sjJIY6fGqJ/cJj+oRH6B4bpOz7AyYFh+oeG6R9MtPUPDTM4PHWPRZ2rs4YFp5Pl9F1y2NiZD5/xfMYud5y+b6olRX3jtKZ8L+e6TJv0MtMP1nGXmWZNqV4m3ZoyWJ1pLfO9FzXyZx9ePP4CsmTKHfkzs1XAKoDzzjsv4mokldLiIppqKmiqqcjK8oZHnMHhEYZGnKHhEQaHnaGREYaG32gfHE7MD42Ex8O0O4y4MzzijHhiK2YktJ2+jSTmT/f1pL4jY/u/0TeZJ7U5JE2f2R46n552T/R5Y/qN9tHnc8Zyfczj478WyX3HLPOMunlz4/j9xjf+YcI0l5lioZOpKZNlptlEqmOh6b7+ZJc5XmNzXeW4z8+mfAdANzA/ab41tJ3m7quB1ZA4CJy/0iRKiV05Om4gkk/5HhD+FLDIzBaYWRlwC7AuzzWIiAh53gJw9yEz+0PgERLDQNe4+wv5rEFERBLyfgzA3R8GHs7364qIyJn0nQAiIjGlABARiSkFgIhITCkARERiSgEgIhJTU/rroM2sF9g5iUXMBvZnqZxsUl2ZUV2ZUV2ZKcS6znf3xok6TekAmCwz60jnK1HzTXVlRnVlRnVlJs51aReQiEhMKQBERGKq0ANgddQFpKC6MqO6MqO6MhPbugr6GICIiKRW6FsAIiKSQkEGgJktM7NtZtZlZnfk+bXnm9ljZrbVzF4ws8+E9j83s24z2xxuNyU95/Oh1m1mdkMOa9thZs+F1+8IbQ1mtt7MOsN9fWg3M7sr1LXFzK7KUU0XJ62TzWZ2xMw+G8X6MrM1ZtZjZs8ntWW8fsxsRejfaWYrclTXX5nZS+G1f2RmdaG9zcxOJq23f0x6ztXh598Vap/UNSxT1JXxzy3bf68p6vpBUk07zGxzaM/n+kr12RDd71jiKkeFcyPxNdOvAAuBMuBZYHEeX78ZuCpMzwReBhYDfw788Tj9F4cay4EFofbiHNW2A5g9pu0vgTvC9B3AV8P0TcD/I3H1uiXAxjz97PYC50exvoD3AFcBz5/r+gEagO3hvj5M1+egrqVASZj+alJdbcn9xiznyVCrhdpvzEFdGf3ccvH3Ol5dYx7/a+B/RLC+Un02RPY7VohbAKcvPO/uA8Dohefzwt33uPvTYfoo8CKJayGnshy4z91PufurQBeJ95Avy4G1YXotcHNS+72e8ARQZ2bNOa7leuAVdz/byX85W1/u/m9A3zivl8n6uQFY7+597n4QWA8sy3Zd7v5Tdx8Ks0+QuLpeSqG2Gnd/whOfIvcmvZes1XUWqX5uWf97PVtd4b/4TwDfP9sycrS+Un02RPY7VogBMN6F58/2AZwzZtYGXAlsDE1/GDbl1oxu5pHfeh34qZltssS1lwGa3H1PmN4LNEVQ16hbOPMPM+r1BZmvnyjW2++S+E9x1AIze8bM/tXM3h3aWkIt+agrk59bvtfXu4F97t6Z1Jb39TXmsyGy37FCDIApwcwKwUOnAAACaElEQVSqgX8GPuvuR4C7gQuAK4A9JDZD8+1d7n4VcCNwu5m9J/nB8J9OJMPCLHGJ0I8APwxNU2F9nSHK9ZOKmX0BGAK+G5r2AOe5+5XA54DvmVlNHkuacj+3MT7Jmf9k5H19jfPZcFq+f8cKMQAmvPB8rplZKYkf8Hfd/UEAd9/n7sPuPgJ8kzd2W+StXnfvDvc9wI9CDftGd+2E+5581xXcCDzt7vtCjZGvryDT9ZO3+szsd4APA58KHxyEXSwHwvQmEvvXLwo1JO8mykld5/Bzy+f6KgF+C/hBUr15XV/jfTYQ4e9YIQZApBeeD/sY7wFedPevJbUn7z//TWB0hMI64BYzKzezBcAiEgefsl1XlZnNHJ0mcRDx+fD6o6MIVgAPJdV1axiJsAQ4nLSZmgtn/GcW9fpKkun6eQRYamb1YffH0tCWVWa2DPgT4CPufiKpvdHMisP0QhLrZ3uo7YiZLQm/o7cmvZds1pXpzy2ff68fAF5y99O7dvK5vlJ9NhDl79hkjmpP1RuJo+cvk0jzL+T5td9FYhNuC7A53G4C/gl4LrSvA5qTnvOFUOs2JjnS4Cx1LSQxwuJZ4IXR9QLMAjYAncDPgIbQbsDfh7qeA9pzuM6qgANAbVJb3tcXiQDaAwyS2K+68lzWD4l98l3hdluO6uoisR949HfsH0Pfj4af72bgaeA3kpbTTuID+RXg7wgngma5rox/btn+ex2vrtD+HeD3x/TN5/pK9dkQ2e+YzgQWEYmpQtwFJCIiaVAAiIjElAJARCSmFAAiIjGlABARiSkFgIhITCkARERiSgEgIhJT/x8gg9FOjzGARQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights,1)\n",
    "        self.w[5] = -100.\n",
    "        self.w[9] = -100.\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)        \n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, gradient_w5, gradient_w9, eta=0.01):\n",
    "        net.w[5] = net.w[5] - eta * gradient_w5\n",
    "        net.w[9] = net.w[9] - eta * gradient_w9\n",
    "        \n",
    "    def train(self, x, y, iterations=100, eta=0.01):\n",
    "        points = []\n",
    "        losses = []\n",
    "        for i in range(iterations):\n",
    "            points.append([net.w[5][0], net.w[9][0]])\n",
    "            z = self.forward(x)\n",
    "            L = self.loss(z, y)\n",
    "            gradient_w, gradient_b = self.gradient(x, y)\n",
    "            gradient_w5 = gradient_w[5][0]\n",
    "            gradient_w9 = gradient_w[9][0]\n",
    "            self.update(gradient_w5, gradient_w9, eta)\n",
    "            losses.append(L)\n",
    "            if i % 50 == 0:\n",
    "                print('iter {}, point {}, loss {}'.format(i, [net.w[5][0], net.w[9][0]], L))\n",
    "        return points, losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "x = train_data[:, :-1]\n",
    "y = train_data[:, -1:]\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "num_iterations=2000\n",
    "# 启动训练\n",
    "points, losses = net.train(x, y, iterations=num_iterations, eta=0.01)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(num_iterations)\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练扩展到全部参数\n",
    "\n",
    "为了能给读者直观的感受，上面演示的梯度下降的过程仅包含$w_5$和$w_9$两个参数，但房价预测的完整模型，必须要对所有参数$w$和$b$进行求解。这需要将Network中的`update`和`train`函数进行修改。由于不再限定参与计算的参数（所有参数均参与计算），修改之后的代码反而更加简洁。实现逻辑：“前向计算输出、根据输出和真实值计算Loss、基于Loss和输入计算梯度、根据梯度更新参数值”四个部分反复执行，直到到损失函数最小。具体代码如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter 9, loss 5.143394325795511\n",
      "iter 19, loss 3.097924194225988\n",
      "iter 29, loss 2.082241020617026\n",
      "iter 39, loss 1.5673801618157397\n",
      "iter 49, loss 1.2966204735077431\n",
      "iter 59, loss 1.1453399043319765\n",
      "iter 69, loss 1.0530155717435201\n",
      "iter 79, loss 0.9902292156463155\n",
      "iter 89, loss 0.9426576903842504\n",
      "iter 99, loss 0.9033048096880774\n",
      "iter 109, loss 0.868732003041364\n",
      "iter 119, loss 0.837229250968144\n",
      "iter 129, loss 0.807927474161227\n",
      "iter 139, loss 0.7803677341465797\n",
      "iter 149, loss 0.7542920908532763\n",
      "iter 159, loss 0.7295420168915829\n",
      "iter 169, loss 0.7060090054240882\n",
      "iter 179, loss 0.6836105084697767\n",
      "iter 189, loss 0.6622781710179412\n",
      "iter 199, loss 0.6419520361168637\n",
      "iter 209, loss 0.622577651786949\n",
      "iter 219, loss 0.6041045903195837\n",
      "iter 229, loss 0.5864856570315078\n",
      "iter 239, loss 0.5696764374763879\n",
      "iter 249, loss 0.5536350125932015\n",
      "iter 259, loss 0.5383217588525027\n",
      "iter 269, loss 0.5236991929680567\n",
      "iter 279, loss 0.509731841376165\n",
      "iter 289, loss 0.4963861247069634\n",
      "iter 299, loss 0.48363025234390233\n",
      "iter 309, loss 0.47143412454019784\n",
      "iter 319, loss 0.45976924072044867\n",
      "iter 329, loss 0.44860861316590983\n",
      "iter 339, loss 0.4379266855659793\n",
      "iter 349, loss 0.4276992560632111\n",
      "iter 359, loss 0.4179034044959738\n",
      "iter 369, loss 0.4085174235863553\n",
      "iter 379, loss 0.39952075384787633\n",
      "iter 389, loss 0.39089392200622347\n",
      "iter 399, loss 0.382618482740513\n",
      "iter 409, loss 0.3746769635645124\n",
      "iter 419, loss 0.36705281267772816\n",
      "iter 429, loss 0.35973034962581096\n",
      "iter 439, loss 0.35269471861856694\n",
      "iter 449, loss 0.3459318443621334\n",
      "iter 459, loss 0.33942839026966587\n",
      "iter 469, loss 0.33317171892221653\n",
      "iter 479, loss 0.3271498546584252\n",
      "iter 489, loss 0.3213514481781961\n",
      "iter 499, loss 0.31576574305173283\n",
      "iter 509, loss 0.3103825440311682\n",
      "iter 519, loss 0.30519218706757245\n",
      "iter 529, loss 0.30018551094136725\n",
      "iter 539, loss 0.29535383041913843\n",
      "iter 549, loss 0.29068891085453674\n",
      "iter 559, loss 0.28618294415539336\n",
      "iter 569, loss 0.28182852604338504\n",
      "iter 579, loss 0.27761863453655344\n",
      "iter 589, loss 0.27354660958874766\n",
      "iter 599, loss 0.2696061338236152\n",
      "iter 609, loss 0.26579121430413205\n",
      "iter 619, loss 0.26209616528184804\n",
      "iter 629, loss 0.25851559187303397\n",
      "iter 639, loss 0.25504437461176843\n",
      "iter 649, loss 0.2516776548326958\n",
      "iter 659, loss 0.2484108208387405\n",
      "iter 669, loss 0.24523949481147198\n",
      "iter 679, loss 0.24215952042409844\n",
      "iter 689, loss 0.2391669511192288\n",
      "iter 699, loss 0.2362580390155805\n",
      "iter 709, loss 0.2334292244097483\n",
      "iter 719, loss 0.2306771258409729\n",
      "iter 729, loss 0.22799853068858245\n",
      "iter 739, loss 0.22539038627340982\n",
      "iter 749, loss 0.22284979143604464\n",
      "iter 759, loss 0.22037398856623477\n",
      "iter 769, loss 0.21796035605914357\n",
      "iter 779, loss 0.2156064011754777\n",
      "iter 789, loss 0.21330975328373866\n",
      "iter 799, loss 0.2110681574640261\n",
      "iter 809, loss 0.20887946845393043\n",
      "iter 819, loss 0.20674164491810018\n",
      "iter 829, loss 0.2046527440240648\n",
      "iter 839, loss 0.20261091630783168\n",
      "iter 849, loss 0.20061440081366638\n",
      "iter 859, loss 0.1986615204933024\n",
      "iter 869, loss 0.19675067785062839\n",
      "iter 879, loss 0.19488035081864621\n",
      "iter 889, loss 0.19304908885621125\n",
      "iter 899, loss 0.1912555092527351\n",
      "iter 909, loss 0.1894982936296714\n",
      "iter 919, loss 0.18777618462820625\n",
      "iter 929, loss 0.18608798277314595\n",
      "iter 939, loss 0.18443254350353405\n",
      "iter 949, loss 0.18280877436103968\n",
      "iter 959, loss 0.18121563232764165\n",
      "iter 969, loss 0.17965212130459232\n",
      "iter 979, loss 0.1781172897250724\n",
      "iter 989, loss 0.1766102282933619\n",
      "iter 999, loss 0.17513006784373505\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAF/NJREFUeJzt3WuMZPdZ5/HfU6du3dWX6ZnuuXjGds/EzgTHAex0wN6AASdLQjYCAQE5LEuACItdWELECiVaraJ9s1qkKCRIUcAKAW2Iwi5ORCLLm7BrEpMA66R9Wd/G9tgz9tw9PZ6e6Xt1XR5enFM93T3dXTUzXVP/U/39SKVT59LVz5lj/+rfT506x9xdAID0yHS6AADAlSG4ASBlCG4ASBmCGwBShuAGgJQhuAEgZQhuAEgZghsAUobgBoCUybbjRYeHh310dLQdLw0AXenxxx8/5+4jrWzbluAeHR3V+Ph4O14aALqSmb3W6ra0SgAgZQhuAEgZghsAUobgBoCUIbgBIGUIbgBIGYIbAFImqOD+k0cO69GXJjpdBgAELajg/tNHX9F3CG4A2FBQwZ3PZrRYq3e6DAAIWljBHWW0WCW4AWAjYQV3luAGgGaCC+4yrRIA2FBYwU2rBACaCiu4aZUAQFNhBXeUUYVWCQBsKKzgZsQNAE2FF9yMuAFgQ2EFNx9OAkBTYQU3rRIAaCq44C4T3ACwoaCCu0CPGwCaaim4zeyjZvacmT1rZl82s2I7iqHHDQDNNQ1uM9sr6fckjbn77ZIiSfe1oxh63ADQXKutkqykHjPLSuqVdKodxXA6IAA01zS43f2kpE9KOibptKSL7v53q7czs/vNbNzMxicmru5mCLkoo1rdVav7Vf08AGwFrbRKhiT9nKT9km6QVDKzX129nbs/4O5j7j42MjJyVcXks3E5tEsAYH2ttEreLemou0+4e0XSVyX9q3YUk48IbgBoppXgPibpLjPrNTOT9C5Jh9pRTKEx4qbPDQDraqXH/ZikByU9IemZ5GceaEcxeYIbAJrKtrKRu39C0ifaXAs9bgBoQVDfnMxHkSSCGwA2ElZwM+IGgKbCDO5arcOVAEC4wgru5HRArhAIAOsLK7hplQBAU0EFd4HgBoCmggruXMR53ADQTFDBTasEAJojuAEgZcIK7qRVUqFVAgDrCiu4s5wOCADNBBXcBYIbAJoiuAEgZYIKbjNTIZtRucJX3gFgPUEFtyQVc5EWCG4AWFdwwV3IZmiVAMAGggtuRtwAsLHggpsRNwBsLLjgZsQNABsLMLgzWqgw4gaA9QQX3IVspHKVETcArCe44GbEDQAbCy64GXEDwMbCC25G3ACwofCCOxtxOiAAbCC44C7muFYJAGwkwOCOtECPGwDWFVxwF7IZVWquWt07XQoABCm44C7mIknizBIAWEdwwb10MwXOLAGANQUX3I0RN31uAFhbgMEdl8S53ACwtuCCu5Clxw0AGwkuuBlxA8DGggvupRE3X8IBgDUFF9xLI26+9g4AawouuBlxA8DGWgpuM9tmZg+a2QtmdsjM7m5XQYy4AWBj2Ra3+4ykb7j7B8wsL6m3XQU1RtwLi4y4AWAtTYPbzAYl3SPp1yXJ3RclLbaroN58HNzztEoAYE2ttEr2S5qQ9Bdm9qSZfd7MSu0qqIfgBoANtRLcWUl3Svqcu98haVbSx1ZvZGb3m9m4mY1PTExcdUHFpFUyR6sEANbUSnCfkHTC3R9L5h9UHOQruPsD7j7m7mMjIyNXX1DGVMxlNL9YverXAIBu1jS43f2MpONmdjBZ9C5Jz7ezqN58llYJAKyj1bNK/qOkLyVnlByR9BvtK0nqyUW0SgBgHS0Ft7s/JWmszbUs6clHmie4AWBNwX1zUopPCaRVAgBrCzK4i7RKAGBdQQZ3bz7SAiNuAFhTsMHNiBsA1hZkcBdzfDgJAOsJMrj5cBIA1hdocGc1xzcnAWBNQQZ3MRdpoVJXve6dLgUAghNkcDcu7brAnd4B4DJBBndPjisEAsB6wgzuxjW5CW4AuEyQwc1dcABgfUEGN60SAFhfmMFNqwQA1hVkcPfm46vNci43AFwuyODuK8Qj7pkywQ0AqwUZ3KVCPOKeLdMqAYDVAg9uRtwAsFqYwZ30uGmVAMDlggzuKGPqzUeMuAFgDUEGtxS3SxhxA8Dlgg3uPoIbANYUbHCXCrRKAGAt4QZ3PsvpgACwhmCDm1YJAKwt2OAuFbKa5SvvAHCZYIO7r5jVzALBDQCrhRvctEoAYE3BBncpn1W5Wle1Vu90KQAQlHCDO7lCIGeWAMBKwQZ3X3KhqRk+oASAFYINbq4QCABrCza4+4txcE8vVDpcCQCEJdjgHujJSZKm5hlxA8By4QZ3MQluRtwAsEKwwT24NOImuAFguWCDu9HjvkhwA8AKLQe3mUVm9qSZPdTOghqKuUiFbEZTfO0dAFa4khH3RyQdalchaxnoydEqAYBVWgpuM9sn6d9I+nx7y1lpsCfHh5MAsEqrI+5PS/pDSdf1wiEDxSw9bgBYpWlwm9n7JZ1198ebbHe/mY2b2fjExMSmFBe3SuhxA8ByrYy43ynpZ83sVUl/LeleM/ur1Ru5+wPuPubuYyMjI5tS3ECRVgkArNY0uN394+6+z91HJd0n6e/d/VfbXpniHjetEgBYKdjzuCVpoCerqfmK3L3TpQBAMK4ouN392+7+/nYVs9pAMae6S7OLXJMbABqCHnE3vvZ+YW6xw5UAQDiCDu5tvXlJ0oU5+twA0BB0cO/oi4P7/CwjbgBoCDq4h5IR9yStEgBYEnRwby8x4gaA1YIO7sGenMykSYIbAJYEHdxRxrStJ6fztEoAYEnQwS1JQ6W8Jmc5qwQAGoIP7h2lPD1uAFgm+OAe6s1zVgkALBN8cG9nxA0AKwQf3EOleMTNhaYAIBZ8cO8o5VWpOTdUAIBE8ME90l+QJE3MLHS4EgAIQ/DBvbO/KEk6O1XucCUAEIbwg3sgHnGfnSa4AUBKQXA3WiVnp2mVAICUguDuL2RVzGVolQBAIvjgNjPt7C/SKgGARPDBLUk7+wuaILgBQFJagnugQI8bABLpCO7+Ij1uAEikIrj3DBY1Xa5qaoHLuwJAKoJ771CPJOnUhfkOVwIAnZeK4L5hWxzcJycJbgBIRXDv28aIGwAaUhHcw30F5aOMThDcAJCO4M5kTHu2FXXqAqcEAkAqgluS9m7r0cnJuU6XAQAdl5rg3jfUo+N8OAkA6Qnu0eGSJqbLmilzJxwAW1tqgvvAcEmS9Oq52Q5XAgCdlZrgHk2C+wjBDWCLS09w72DEDQBSioK7mIt0w2BRRwluAFtcaoJbkg6M9OmViZlOlwEAHZWq4D64u18vvT6tWt07XQoAdEzT4DazG83sW2b2vJk9Z2YfuR6FreUtu/u1UKnrtTdolwDYuloZcVcl/YG73ybpLkm/Y2a3tbestf3AngFJ0qHT05349QAQhKbB7e6n3f2J5Pm0pEOS9ra7sLXcsrNPUcb0wpmpTvx6AAjCFfW4zWxU0h2SHmtHMc0Uc5HeNFLSc6cIbgBbV8vBbWZ9kr4i6ffd/bLkNLP7zWzczMYnJiY2s8YVfmjfNj11/ILc+YASwNbUUnCbWU5xaH/J3b+61jbu/oC7j7n72MjIyGbWuMKdNw/p/OyiXnuDKwUC2JpaOavEJP25pEPu/qn2l7SxO27aJkl68vhkhysBgM5oZcT9Tkn/TtK9ZvZU8nhfm+ta1607+9VXyOr7rxLcALambLMN3P27kuw61NKSKGO668B2fffwuU6XAgAdkapvTjb8xJtHdOz8HBecArAlpTK473lz/OHnoy+17+wVAAhVKoP75h0l3byjl+AGsCWlMrgl6d0/sEvfPXxOF+cqnS4FAK6r1Ab3z9+xV4u1uh565lSnSwGA6yq1wf3WGwZ0y84+/e2TJztdCgBcV6kNbjPTL9y5V99/dZKLTgHYUlIb3JL0wXfcpN58pAcePdLpUgDgukl1cA+V8rrvHTfp6///lI6f59olALaGVAe3JP3WPfuVjUz/7eFDnS4FAK6L1Af3nsEe/e5P3aL//ewZffvFs50uBwDaLvXBLUm/dc8B3bKzT//pb57W2emFTpcDAG3VFcFdyEb67K/cqZlyRf/+r57Q/GKt0yUBQNt0RXBL0sHd/frUL/+wnjw2qfu/OK6ZcrXTJQFAW3RNcEvS+962R3/0iz+of3rlDX3gc//EmSYAulJXBbck/dLYjfrCr79DJyfn9Z5P/4O++M+vqlbn/pQAukfXBbcUX6/7Gx+9R2+/eUj/5WvP6b2f/gc9/Mxp1QlwAF2gK4NbkvZu69H/+M0f0Wd/5U7V3fUfvvSEfvKT39afPvqKzs2UO10eAFw1c9/8UejY2JiPj49v+uterVrd9fAzp/XF//eavnf0vDIm3XVgh37mbXv0nrfu0s7+YqdLBLDFmdnj7j7W0rZbIbiXO/z6tL721Ck9/OxpHZmIb3321hsG9GO3DuvHbxnR2OiQirmow1UC2GoI7ha4uw6fndHfPXdG3zl8Tk8cm1Sl5ipkM3r7zUMau3lIbx/drjtu2qaBYq7T5QLocgT3VZgtV/XY0Tf0ncPn9L2j53Xo9JTqLplJB3f1a2x0SHfeNKS37R3UgZE+RZlgbnwPoAsQ3JtgplzVU8cuaPy183r8tUk9eezC0pd6evORbtszoNv3Duptewf1g/sIcwDXhuBug1rd9crEjJ45cVHPnLyoZ09e1HOnpjRfib9e35uPdHB3v96yu18Hd/Xrzcl0R1+hw5UDSAOC+zqp1V1HJmb0dBLmL5yZ0otnpjW57AbGw30FHdzdp4O7BnRwd59u2dmvA8MlDZXyHawcQGiuJLiz7S6mm0UZ0627+nXrrn794tv3SYo/9JyYKevFM9NLj5den9aXv3dsaXQuSdt6c9o/XNKB4T4dGClp//ClB2e1ANgIwb3JzEw7+4va2V/Uj986srS8XncdOz+nI+dmdGRiVkfOzeroxKz+8eVz+soTJ1a8xg2DRd24vTd+DPXqxu092pdMd/UXlaGXDmxpBPd1ksmYRodLGh0u6d63rFw3W67q6LnZFY9j5+f0ncMTen1q5bc881FGe4d6tG8oDvN9Qz3aM1jU7sGidg/E0948hxXoZvwfHoBSIavb9w7q9r2Dl61bqNR08sK8TkzO6/j5OR2fnNOJyXmdOD+nb546o/Ozi5f9zEAxqz2DPSvCvPEY6StopL+gHaW8slHXXvEA6GoEd+CKuUhvGunTm0b61lw/v1jTmakFnb44r9enFnT64oLONB5TCzp0ekoTM2Wt9Rn0UG9Ow32F+NFf0HBfXsN9BY30FTTcn19at72Up+8OBITgTrmefLT0oeZ6KrW6zk6XdebivCamy5qYWdS56bLOzTQei3r6xAWdmy5rdp27BxVzGQ315rWtN6+h3lzyfOV0qJRL1sfbDBRz9OOBNiC4t4BclNHebT3au62n6bbzizWdmylrYqachPuiJucWdWFuUZNzlaXpoTNTupDMr3e1XDOpL5/VQE9O/cWs+otZDRQbz3Ma6Imny5cP9OQ0ULy0vCcXyYzwB5YjuLFCTz5aOqOlFfW6a3qhqsm5RsBXkucVXZyvaHqhoqn5ajxdqOjM1IIOn61qaqGi6YVq05tcmEm9uUilQlalQla9+UilfFalQqTeQlalfKTefFZ9hax6C411yfJkWirEbwDFXBRP8xnlowxvCEgtghvXJJMxDfbmNNib06jWb9esxd01t1jT9MKlYJ9aqGpqvpIsq2p+sarZxZpmy/F0rlzVTLmqczOLmj0/p7lyY1113ZH/WsyknkaQ5yIVc5mlYO/JRypk42kxm1FPPl5eaKxvbJuPlI8yKuQyykdRMs0on82okI2n8fMono8ytI6wKQhudIyZLY2kdw9e2zXR3V3laj0O8XJNs4tVzS0mz8tVzVdqWqjUk2n8mF+sLS1fWlapLb0xLF82v1hTuVq/5n3ORZaEfbROyGeUbwR9NqNC8saQizLKZjLKZU25TDKfvFY2MuWijHLJNBtllF/2PLe0PqNsxpTPxtPGslxkyc/Er5XNGH+NBI7gRlcws2TkHGnH2ifgXLN6PX5zmF8W6IvVusrVejKN51csq9VVrtS0WKs33XaxVle5UtfF+cqKbcrVuiq1uqo112Itft6GK1WssPQmsBT0GUUZUzayeJoxRZl4fSPs4+WZZesb22eWrW9sv2q7xuuteP3G9qu3XTZ/2e+WMhbPZyz5/WbKZOJplFn5PMo01uvSzyTLQ37zIriBFmUyFrdN8p0/NbJWd1VqlwK9UqurUndVqnVV63UtVl3Vej3ZxlcE/9L2ybp4+7qqyc9X6o3tL/1spVZXrS7V6vF28e/3FfPVumu+Ukvm499TW7auWlu5bTytL71WaMx0Weg3HvGbw7L1yWO4VND/+u27215bS8FtZu+V9BlJkaTPu/t/b2tVADYUB0XUVefX15cFeqVeV612ecAvzdcuD/5a3VVzV72+8nm17qr7pTeR+Hn8JhRvF//uWrLNZT+z9JpS3Ru/89LPNKbVuqu/cH3Gwk1/i5lFkj4r6V9LOiHp+2b2dXd/vt3FAdg6MhlTPvnwtkfd84bUDq185/lHJL3s7kfcfVHSX0v6ufaWBQBYTyvBvVfS8WXzJ5JlAIAO2LSrDJnZ/WY2bmbjExMTm/WyAIBVWgnuk5JuXDa/L1m2grs/4O5j7j42MjKyejUAYJO0Etzfl3Srme03s7yk+yR9vb1lAQDW0/SsEnevmtnvSvqm4tMBv+Duz7W9MgDAmlo66dDdH5b0cJtrAQC0gFugAEDKmLfhogdmNiHptav88WFJ5zaxnDRgn7cG9rn7Xcv+3uzuLZ3Z0ZbgvhZmNu7uY52u43pin7cG9rn7Xa/9pVUCAClDcANAyoQY3A90uoAOYJ+3Bva5+12X/Q2uxw0A2FiII24AwAaCCW4ze6+ZvWhmL5vZxzpdz2YxsxvN7Ftm9ryZPWdmH0mWbzez/2Nmh5PpULLczOxPkn+Hp83szs7uwdUzs8jMnjSzh5L5/Wb2WLJv/zO5hILMrJDMv5ysH+1k3VfLzLaZ2YNm9oKZHTKzu7v9OJvZR5P/rp81sy+bWbHbjrOZfcHMzprZs8uWXfFxNbMPJdsfNrMPXUtNQQT3sps1/Iyk2yR90Mxu62xVm6Yq6Q/c/TZJd0n6nWTfPibpEXe/VdIjybwU/xvcmjzul/S561/ypvmIpEPL5v9I0h+7+y2SJiV9OFn+YUmTyfI/TrZLo89I+oa7v0XSDyne9649zma2V9LvSRpz99sVXxLjPnXfcf5LSe9dteyKjquZbZf0CUk/qvgeB59ohP1VcfeOPyTdLemby+Y/Lunjna6rTfv6NcV3E3pR0p5k2R5JLybP/0zSB5dtv7Rdmh6KryL5iKR7JT0kyRR/MSG7+pgrvg7O3cnzbLKddXofrnB/ByUdXV13Nx9nXbpW//bkuD0k6T3deJwljUp69mqPq6QPSvqzZctXbHeljyBG3NoiN2tI/jS8Q9Jjkna5++lk1RlJu5Ln3fJv8WlJfyipnszvkHTB3avJ/PL9WtrnZP3FZPs02S9pQtJfJO2hz5tZSV18nN39pKRPSjom6bTi4/a4uvs4N1zpcd3U4x1KcHc9M+uT9BVJv+/uU8vXefwW3DWn95jZ+yWddffHO13LdZSVdKekz7n7HZJmdenPZ0ldeZyHFN/GcL+kGySVdHlLoet14riGEtwt3awhrcwspzi0v+TuX00Wv25me5L1eySdTZZ3w7/FOyX9rJm9qvgepfcq7v9uM7PGFSmX79fSPifrByW9cT0L3gQnJJ1w98eS+QcVB3k3H+d3Szrq7hPuXpH0VcXHvpuPc8OVHtdNPd6hBHfX3qzBzEzSn0s65O6fWrbq65Ianyx/SHHvu7H815JPp++SdHHZn2Sp4O4fd/d97j6q+Fj+vbv/W0nfkvSBZLPV+9z4t/hAsn2qRqbufkbScTM7mCx6l6Tn1cXHWXGL5C4z603+O2/sc9ce52Wu9Lh+U9JPm9lQ8pfKTyfLrk6nm/7LmvXvk/SSpFck/edO17OJ+/Vjiv+MelrSU8njfYp7e49IOizp/0ranmxvis+weUXSM4o/se/4flzD/v+kpIeS5wckfU/Sy5L+RlIhWV5M5l9O1h/odN1Xua8/LGk8OdZ/K2mo24+zpP8q6QVJz0r6oqRCtx1nSV9W3MOvKP7L6sNXc1wl/Way7y9L+o1rqYlvTgJAyoTSKgEAtIjgBoCUIbgBIGUIbgBIGYIbAFKG4AaAlCG4ASBlCG4ASJl/ATh8K97JO6m6AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)        \n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, gradient_w, gradient_b, eta = 0.01):\n",
    "        self.w = self.w - eta * gradient_w\n",
    "        self.b = self.b - eta * gradient_b\n",
    "        \n",
    "    def train(self, x, y, iterations=100, eta=0.01):\n",
    "        losses = []\n",
    "        for i in range(iterations):\n",
    "            z = self.forward(x)\n",
    "            L = self.loss(z, y)\n",
    "            gradient_w, gradient_b = self.gradient(x, y)\n",
    "            self.update(gradient_w, gradient_b, eta)\n",
    "            losses.append(L)\n",
    "            if (i+1) % 10 == 0:\n",
    "                print('iter {}, loss {}'.format(i, L))\n",
    "        return losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "x = train_data[:, :-1]\n",
    "y = train_data[:, -1:]\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "num_iterations=1000\n",
    "# 启动训练\n",
    "losses = net.train(x,y, iterations=num_iterations, eta=0.01)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(num_iterations)\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 随机梯度下降法（ Stochastic Gradient Descent）\n",
    "\n",
    "在上述程序中，每次损失函数和梯度计算都是基于数据集中的全量数据。对于波士顿房价预测任务数据集而言，样本数比较少，只有404个。但在实际问题中，数据集往往非常大，如果每次都使用全量数据进行计算，效率非常低，通俗地说就是“杀鸡焉用牛刀”。由于参数每次只沿着梯度反方向更新一点点，因此方向并不需要那么精确。一个合理的解决方案是每次从总的数据集中随机抽取出小部分数据来代表整体，基于这部分数据计算梯度和损失来更新参数，这种方法被称作随机梯度下降法（Stochastic Gradient Descent，SGD），核心概念如下：\n",
    "\n",
    "* mini-batch：每次迭代时抽取出来的一批数据被称为一个mini-batch。\n",
    "* batch_size：一个mini-batch所包含的样本数目称为batch_size。\n",
    "* epoch：当程序迭代的时候，按mini-batch逐渐抽取出样本，当把整个数据集都遍历到了的时候，则完成了一轮训练，也叫一个epoch。启动训练时，可以将训练的轮数num_epochs和batch_size作为参数传入。\n",
    "\n",
    "下面结合程序介绍具体的实现过程，涉及到数据处理和训练过程两部分代码的修改。\n",
    "\n",
    "#### **数据处理代码修改**\n",
    "\n",
    "数据处理需要实现拆分数据批次和样本乱序（为了实现随机抽样的效果）两个功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(404, 14)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "train_data中一共包含404条数据，如果batch_size=10，即取前0-9号样本作为第一个mini-batch，命名train_data1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 14)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data1 = train_data[0:10]\n",
    "train_data1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用train_data1的数据（0-9号样本）计算梯度并更新网络参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4.497480200683046]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = Network(13)\n",
    "x = train_data1[:, :-1]\n",
    "y = train_data1[:, -1:]\n",
    "loss = net.train(x, y, iterations=1, eta=0.01)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "再取出10-19号样本作为第二个mini-batch，计算梯度并更新网络参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[5.849682302465982]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data2 = train_data[10:20]\n",
    "x = train_data2[:, :-1]\n",
    "y = train_data2[:, -1:]\n",
    "loss = net.train(x, y, iterations=1, eta=0.01)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "按此方法不断的取出新的mini-batch，并逐渐更新网络参数。\n",
    "\n",
    "接下来，将train_data分成大小为batch_size的多个mini_batch，如下代码所示：将train_data分成 $\\frac{404}{10} + 1 = 41$ 个 mini_batch，其中前40个mini_batch，每个均含有10个样本，最后一个mini_batch只含有4个样本。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total number of mini_batches is  41\n",
      "first mini_batch shape  (10, 14)\n",
      "last mini_batch shape  (4, 14)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 10\n",
    "n = len(train_data)\n",
    "mini_batches = [train_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "print('total number of mini_batches is ', len(mini_batches))\n",
    "print('first mini_batch shape ', mini_batches[0].shape)\n",
    "print('last mini_batch shape ', mini_batches[-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "另外，这里是按顺序读取mini_batch，而SGD里面是随机抽取一部分样本代表总体。为了实现随机抽样的效果，我们先将train_data里面的样本顺序随机打乱，然后再抽取mini_batch。随机打乱样本顺序，需要用到`np.random.shuffle`函数，下面先介绍它的用法。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "通过大量实验发现，模型对最后出现的数据印象更加深刻。训练数据导入后，越接近模型训练结束，最后几个批次数据对模型参数的影响越大。为了避免模型记忆影响训练效果，需要进行样本乱序操作。\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before shuffle [ 1  2  3  4  5  6  7  8  9 10 11 12]\n",
      "after shuffle [ 7  2 11  3  8  6 12  1  4  5 10  9]\n"
     ]
    }
   ],
   "source": [
    "# 新建一个array\n",
    "a = np.array([1,2,3,4,5,6,7,8,9,10,11,12])\n",
    "print('before shuffle', a)\n",
    "np.random.shuffle(a)\n",
    "print('after shuffle', a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "多次运行上面的代码，可以发现每次执行shuffle函数后的数字顺序均不同。\n",
    "上面举的是一个1维数组乱序的案例，我们再观察下2维数组乱序后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before shuffle\n",
      " [[ 1  2]\n",
      " [ 3  4]\n",
      " [ 5  6]\n",
      " [ 7  8]\n",
      " [ 9 10]\n",
      " [11 12]]\n",
      "after shuffle\n",
      " [[ 1  2]\n",
      " [ 3  4]\n",
      " [ 5  6]\n",
      " [ 9 10]\n",
      " [11 12]\n",
      " [ 7  8]]\n"
     ]
    }
   ],
   "source": [
    "# 新建一个array\n",
    "a = np.array([1,2,3,4,5,6,7,8,9,10,11,12])\n",
    "a = a.reshape([6, 2])\n",
    "print('before shuffle\\n', a)\n",
    "np.random.shuffle(a)\n",
    "print('after shuffle\\n', a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "观察运行结果可发现，数组的元素在第0维被随机打乱，但第1维的顺序保持不变。例如数字2仍然紧挨在数字1的后面，数字8仍然紧挨在数字7的后面，而第二维的[3, 4]并不排在[1, 2]的后面。将这部分实现SGD算法的代码集成到Network类中的`train`函数中，最终的完整代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "\n",
    "# 打乱样本顺序\n",
    "np.random.shuffle(train_data)\n",
    "\n",
    "# 将train_data分成多个mini_batch\n",
    "batch_size = 10\n",
    "n = len(train_data)\n",
    "mini_batches = [train_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "\n",
    "# 依次使用每个mini_batch的数据\n",
    "for mini_batch in mini_batches:\n",
    "    x = mini_batch[:, :-1]\n",
    "    y = mini_batch[:, -1:]\n",
    "    loss = net.train(x, y, iterations=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **训练过程代码修改**\n",
    "\n",
    "将每个随机抽取的mini-batch数据输入到模型中用于参数训练。训练过程的核心是两层循环：\n",
    "\n",
    "1. 第一层循环，代表样本集合要被训练遍历几次，称为“epoch”，代码如下：\n",
    "\n",
    "`for epoch_id in range(num_epochs):`\n",
    "\n",
    "2. 第二层循环，代表每次遍历时，样本集合被拆分成的多个批次，需要全部执行训练，称为“iter (iteration)”，代码如下：\n",
    "\n",
    "`for iter_id,mini_batch in emumerate(mini_batches):`\n",
    "\n",
    "在两层循环的内部是经典的四步训练流程：前向计算->计算损失->计算梯度->更新参数，这与大家之前所学是一致的，代码如下：\n",
    "\n",
    "                x = mini_batch[:, :-1]\n",
    "                y = mini_batch[:, -1:]\n",
    "                a = self.forward(x)  #前向计算\n",
    "                loss = self.loss(a, y)  #计算损失\n",
    "                gradient_w, gradient_b = self.gradient(x, y)  #计算梯度\n",
    "                self.update(gradient_w, gradient_b, eta)  #更新参数\n",
    "\n",
    "\n",
    "将两部分改写的代码集成到Network类中的`train`函数中，最终的实现如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 / iter   0, loss = 1.0281\n",
      "Epoch   0 / iter   1, loss = 0.5048\n",
      "Epoch   0 / iter   2, loss = 0.6382\n",
      "Epoch   0 / iter   3, loss = 0.5168\n",
      "Epoch   0 / iter   4, loss = 0.1951\n",
      "Epoch   1 / iter   0, loss = 0.6281\n",
      "Epoch   1 / iter   1, loss = 0.4611\n",
      "Epoch   1 / iter   2, loss = 0.4520\n",
      "Epoch   1 / iter   3, loss = 0.3961\n",
      "Epoch   1 / iter   4, loss = 0.1381\n",
      "Epoch   2 / iter   0, loss = 0.5642\n",
      "Epoch   2 / iter   1, loss = 0.4250\n",
      "Epoch   2 / iter   2, loss = 0.4480\n",
      "Epoch   2 / iter   3, loss = 0.3881\n",
      "Epoch   2 / iter   4, loss = 0.1884\n",
      "Epoch   3 / iter   0, loss = 0.3921\n",
      "Epoch   3 / iter   1, loss = 0.5582\n",
      "Epoch   3 / iter   2, loss = 0.3759\n",
      "Epoch   3 / iter   3, loss = 0.3849\n",
      "Epoch   3 / iter   4, loss = 0.1425\n",
      "Epoch   4 / iter   0, loss = 0.3821\n",
      "Epoch   4 / iter   1, loss = 0.4382\n",
      "Epoch   4 / iter   2, loss = 0.3864\n",
      "Epoch   4 / iter   3, loss = 0.4314\n",
      "Epoch   4 / iter   4, loss = 0.0471\n",
      "Epoch   5 / iter   0, loss = 0.4264\n",
      "Epoch   5 / iter   1, loss = 0.3829\n",
      "Epoch   5 / iter   2, loss = 0.3179\n",
      "Epoch   5 / iter   3, loss = 0.4149\n",
      "Epoch   5 / iter   4, loss = 0.1581\n",
      "Epoch   6 / iter   0, loss = 0.3148\n",
      "Epoch   6 / iter   1, loss = 0.3532\n",
      "Epoch   6 / iter   2, loss = 0.4195\n",
      "Epoch   6 / iter   3, loss = 0.3272\n",
      "Epoch   6 / iter   4, loss = 1.2465\n",
      "Epoch   7 / iter   0, loss = 0.3166\n",
      "Epoch   7 / iter   1, loss = 0.2810\n",
      "Epoch   7 / iter   2, loss = 0.4126\n",
      "Epoch   7 / iter   3, loss = 0.3309\n",
      "Epoch   7 / iter   4, loss = 0.2255\n",
      "Epoch   8 / iter   0, loss = 0.2555\n",
      "Epoch   8 / iter   1, loss = 0.3678\n",
      "Epoch   8 / iter   2, loss = 0.3342\n",
      "Epoch   8 / iter   3, loss = 0.3806\n",
      "Epoch   8 / iter   4, loss = 0.0570\n",
      "Epoch   9 / iter   0, loss = 0.3532\n",
      "Epoch   9 / iter   1, loss = 0.3973\n",
      "Epoch   9 / iter   2, loss = 0.1945\n",
      "Epoch   9 / iter   3, loss = 0.2839\n",
      "Epoch   9 / iter   4, loss = 0.1604\n",
      "Epoch  10 / iter   0, loss = 0.3414\n",
      "Epoch  10 / iter   1, loss = 0.2774\n",
      "Epoch  10 / iter   2, loss = 0.3439\n",
      "Epoch  10 / iter   3, loss = 0.2103\n",
      "Epoch  10 / iter   4, loss = 0.0959\n",
      "Epoch  11 / iter   0, loss = 0.3004\n",
      "Epoch  11 / iter   1, loss = 0.2497\n",
      "Epoch  11 / iter   2, loss = 0.2827\n",
      "Epoch  11 / iter   3, loss = 0.2987\n",
      "Epoch  11 / iter   4, loss = 0.0316\n",
      "Epoch  12 / iter   0, loss = 0.2509\n",
      "Epoch  12 / iter   1, loss = 0.2535\n",
      "Epoch  12 / iter   2, loss = 0.2944\n",
      "Epoch  12 / iter   3, loss = 0.2889\n",
      "Epoch  12 / iter   4, loss = 0.0547\n",
      "Epoch  13 / iter   0, loss = 0.2792\n",
      "Epoch  13 / iter   1, loss = 0.2137\n",
      "Epoch  13 / iter   2, loss = 0.2427\n",
      "Epoch  13 / iter   3, loss = 0.2986\n",
      "Epoch  13 / iter   4, loss = 0.3861\n",
      "Epoch  14 / iter   0, loss = 0.3261\n",
      "Epoch  14 / iter   1, loss = 0.2123\n",
      "Epoch  14 / iter   2, loss = 0.1837\n",
      "Epoch  14 / iter   3, loss = 0.2968\n",
      "Epoch  14 / iter   4, loss = 0.0620\n",
      "Epoch  15 / iter   0, loss = 0.2402\n",
      "Epoch  15 / iter   1, loss = 0.2823\n",
      "Epoch  15 / iter   2, loss = 0.2574\n",
      "Epoch  15 / iter   3, loss = 0.1833\n",
      "Epoch  15 / iter   4, loss = 0.0637\n",
      "Epoch  16 / iter   0, loss = 0.1889\n",
      "Epoch  16 / iter   1, loss = 0.1998\n",
      "Epoch  16 / iter   2, loss = 0.2031\n",
      "Epoch  16 / iter   3, loss = 0.3219\n",
      "Epoch  16 / iter   4, loss = 0.1373\n",
      "Epoch  17 / iter   0, loss = 0.2042\n",
      "Epoch  17 / iter   1, loss = 0.2070\n",
      "Epoch  17 / iter   2, loss = 0.2651\n",
      "Epoch  17 / iter   3, loss = 0.2137\n",
      "Epoch  17 / iter   4, loss = 0.0138\n",
      "Epoch  18 / iter   0, loss = 0.1794\n",
      "Epoch  18 / iter   1, loss = 0.1575\n",
      "Epoch  18 / iter   2, loss = 0.2554\n",
      "Epoch  18 / iter   3, loss = 0.2531\n",
      "Epoch  18 / iter   4, loss = 0.2192\n",
      "Epoch  19 / iter   0, loss = 0.1779\n",
      "Epoch  19 / iter   1, loss = 0.2072\n",
      "Epoch  19 / iter   2, loss = 0.2140\n",
      "Epoch  19 / iter   3, loss = 0.2513\n",
      "Epoch  19 / iter   4, loss = 0.0673\n",
      "Epoch  20 / iter   0, loss = 0.1634\n",
      "Epoch  20 / iter   1, loss = 0.1887\n",
      "Epoch  20 / iter   2, loss = 0.2515\n",
      "Epoch  20 / iter   3, loss = 0.1924\n",
      "Epoch  20 / iter   4, loss = 0.0926\n",
      "Epoch  21 / iter   0, loss = 0.1583\n",
      "Epoch  21 / iter   1, loss = 0.2319\n",
      "Epoch  21 / iter   2, loss = 0.1550\n",
      "Epoch  21 / iter   3, loss = 0.2092\n",
      "Epoch  21 / iter   4, loss = 0.1959\n",
      "Epoch  22 / iter   0, loss = 0.2414\n",
      "Epoch  22 / iter   1, loss = 0.1522\n",
      "Epoch  22 / iter   2, loss = 0.1719\n",
      "Epoch  22 / iter   3, loss = 0.1829\n",
      "Epoch  22 / iter   4, loss = 0.2748\n",
      "Epoch  23 / iter   0, loss = 0.1861\n",
      "Epoch  23 / iter   1, loss = 0.1830\n",
      "Epoch  23 / iter   2, loss = 0.1606\n",
      "Epoch  23 / iter   3, loss = 0.2351\n",
      "Epoch  23 / iter   4, loss = 0.1479\n",
      "Epoch  24 / iter   0, loss = 0.1678\n",
      "Epoch  24 / iter   1, loss = 0.2080\n",
      "Epoch  24 / iter   2, loss = 0.1471\n",
      "Epoch  24 / iter   3, loss = 0.1747\n",
      "Epoch  24 / iter   4, loss = 0.1607\n",
      "Epoch  25 / iter   0, loss = 0.1162\n",
      "Epoch  25 / iter   1, loss = 0.2067\n",
      "Epoch  25 / iter   2, loss = 0.1692\n",
      "Epoch  25 / iter   3, loss = 0.1757\n",
      "Epoch  25 / iter   4, loss = 0.0125\n",
      "Epoch  26 / iter   0, loss = 0.1707\n",
      "Epoch  26 / iter   1, loss = 0.1898\n",
      "Epoch  26 / iter   2, loss = 0.1409\n",
      "Epoch  26 / iter   3, loss = 0.1501\n",
      "Epoch  26 / iter   4, loss = 0.1002\n",
      "Epoch  27 / iter   0, loss = 0.1590\n",
      "Epoch  27 / iter   1, loss = 0.1801\n",
      "Epoch  27 / iter   2, loss = 0.1578\n",
      "Epoch  27 / iter   3, loss = 0.1257\n",
      "Epoch  27 / iter   4, loss = 0.7750\n",
      "Epoch  28 / iter   0, loss = 0.1573\n",
      "Epoch  28 / iter   1, loss = 0.1224\n",
      "Epoch  28 / iter   2, loss = 0.1353\n",
      "Epoch  28 / iter   3, loss = 0.1862\n",
      "Epoch  28 / iter   4, loss = 0.5305\n",
      "Epoch  29 / iter   0, loss = 0.1981\n",
      "Epoch  29 / iter   1, loss = 0.1114\n",
      "Epoch  29 / iter   2, loss = 0.1414\n",
      "Epoch  29 / iter   3, loss = 0.1856\n",
      "Epoch  29 / iter   4, loss = 0.0268\n",
      "Epoch  30 / iter   0, loss = 0.0984\n",
      "Epoch  30 / iter   1, loss = 0.1528\n",
      "Epoch  30 / iter   2, loss = 0.1637\n",
      "Epoch  30 / iter   3, loss = 0.1532\n",
      "Epoch  30 / iter   4, loss = 0.0846\n",
      "Epoch  31 / iter   0, loss = 0.1433\n",
      "Epoch  31 / iter   1, loss = 0.1643\n",
      "Epoch  31 / iter   2, loss = 0.1202\n",
      "Epoch  31 / iter   3, loss = 0.1215\n",
      "Epoch  31 / iter   4, loss = 0.2182\n",
      "Epoch  32 / iter   0, loss = 0.1567\n",
      "Epoch  32 / iter   1, loss = 0.1420\n",
      "Epoch  32 / iter   2, loss = 0.1073\n",
      "Epoch  32 / iter   3, loss = 0.1496\n",
      "Epoch  32 / iter   4, loss = 0.0846\n",
      "Epoch  33 / iter   0, loss = 0.1420\n",
      "Epoch  33 / iter   1, loss = 0.1369\n",
      "Epoch  33 / iter   2, loss = 0.0962\n",
      "Epoch  33 / iter   3, loss = 0.1480\n",
      "Epoch  33 / iter   4, loss = 0.0687\n",
      "Epoch  34 / iter   0, loss = 0.1234\n",
      "Epoch  34 / iter   1, loss = 0.1028\n",
      "Epoch  34 / iter   2, loss = 0.1407\n",
      "Epoch  34 / iter   3, loss = 0.1528\n",
      "Epoch  34 / iter   4, loss = 0.0390\n",
      "Epoch  35 / iter   0, loss = 0.1113\n",
      "Epoch  35 / iter   1, loss = 0.1289\n",
      "Epoch  35 / iter   2, loss = 0.1733\n",
      "Epoch  35 / iter   3, loss = 0.0892\n",
      "Epoch  35 / iter   4, loss = 0.0456\n",
      "Epoch  36 / iter   0, loss = 0.1358\n",
      "Epoch  36 / iter   1, loss = 0.0782\n",
      "Epoch  36 / iter   2, loss = 0.1475\n",
      "Epoch  36 / iter   3, loss = 0.1294\n",
      "Epoch  36 / iter   4, loss = 0.0442\n",
      "Epoch  37 / iter   0, loss = 0.1136\n",
      "Epoch  37 / iter   1, loss = 0.0954\n",
      "Epoch  37 / iter   2, loss = 0.1542\n",
      "Epoch  37 / iter   3, loss = 0.1262\n",
      "Epoch  37 / iter   4, loss = 0.0452\n",
      "Epoch  38 / iter   0, loss = 0.1277\n",
      "Epoch  38 / iter   1, loss = 0.1361\n",
      "Epoch  38 / iter   2, loss = 0.1103\n",
      "Epoch  38 / iter   3, loss = 0.0920\n",
      "Epoch  38 / iter   4, loss = 0.4119\n",
      "Epoch  39 / iter   0, loss = 0.1054\n",
      "Epoch  39 / iter   1, loss = 0.1165\n",
      "Epoch  39 / iter   2, loss = 0.1334\n",
      "Epoch  39 / iter   3, loss = 0.1240\n",
      "Epoch  39 / iter   4, loss = 0.0672\n",
      "Epoch  40 / iter   0, loss = 0.1218\n",
      "Epoch  40 / iter   1, loss = 0.0982\n",
      "Epoch  40 / iter   2, loss = 0.1077\n",
      "Epoch  40 / iter   3, loss = 0.1062\n",
      "Epoch  40 / iter   4, loss = 0.4781\n",
      "Epoch  41 / iter   0, loss = 0.1541\n",
      "Epoch  41 / iter   1, loss = 0.1049\n",
      "Epoch  41 / iter   2, loss = 0.0979\n",
      "Epoch  41 / iter   3, loss = 0.1042\n",
      "Epoch  41 / iter   4, loss = 0.0397\n",
      "Epoch  42 / iter   0, loss = 0.0996\n",
      "Epoch  42 / iter   1, loss = 0.1031\n",
      "Epoch  42 / iter   2, loss = 0.1294\n",
      "Epoch  42 / iter   3, loss = 0.0980\n",
      "Epoch  42 / iter   4, loss = 0.1135\n",
      "Epoch  43 / iter   0, loss = 0.1521\n",
      "Epoch  43 / iter   1, loss = 0.1088\n",
      "Epoch  43 / iter   2, loss = 0.1089\n",
      "Epoch  43 / iter   3, loss = 0.0775\n",
      "Epoch  43 / iter   4, loss = 0.1444\n",
      "Epoch  44 / iter   0, loss = 0.0827\n",
      "Epoch  44 / iter   1, loss = 0.0875\n",
      "Epoch  44 / iter   2, loss = 0.1428\n",
      "Epoch  44 / iter   3, loss = 0.1002\n",
      "Epoch  44 / iter   4, loss = 0.0352\n",
      "Epoch  45 / iter   0, loss = 0.0917\n",
      "Epoch  45 / iter   1, loss = 0.1193\n",
      "Epoch  45 / iter   2, loss = 0.0933\n",
      "Epoch  45 / iter   3, loss = 0.1044\n",
      "Epoch  45 / iter   4, loss = 0.0064\n",
      "Epoch  46 / iter   0, loss = 0.1020\n",
      "Epoch  46 / iter   1, loss = 0.0913\n",
      "Epoch  46 / iter   2, loss = 0.0882\n",
      "Epoch  46 / iter   3, loss = 0.1170\n",
      "Epoch  46 / iter   4, loss = 0.0330\n",
      "Epoch  47 / iter   0, loss = 0.0696\n",
      "Epoch  47 / iter   1, loss = 0.0996\n",
      "Epoch  47 / iter   2, loss = 0.0948\n",
      "Epoch  47 / iter   3, loss = 0.1109\n",
      "Epoch  47 / iter   4, loss = 0.5095\n",
      "Epoch  48 / iter   0, loss = 0.0929\n",
      "Epoch  48 / iter   1, loss = 0.1220\n",
      "Epoch  48 / iter   2, loss = 0.1150\n",
      "Epoch  48 / iter   3, loss = 0.0917\n",
      "Epoch  48 / iter   4, loss = 0.0968\n",
      "Epoch  49 / iter   0, loss = 0.0732\n",
      "Epoch  49 / iter   1, loss = 0.0808\n",
      "Epoch  49 / iter   2, loss = 0.0896\n",
      "Epoch  49 / iter   3, loss = 0.1306\n",
      "Epoch  49 / iter   4, loss = 0.1896\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztvXmcZFV5//8599be+zY9W/fsAwwMA+MwgCCLYhwIQvwaEdRoBCWJkujXxKgxQWNi8lMSNSbGiAZwBVdkfooiIMjmMAwwzL6vPUvvS3Xtde/5/nHvOXepqu7q7qquW9XP+/Wa11RX3646t5bPee7nec5zGOccBEEQRG2hVHoABEEQROkhcScIgqhBSNwJgiBqEBJ3giCIGoTEnSAIogYhcScIgqhBSNwJgiBqEBJ3giCIGoTEnSAIogbxVeqJ29vb+dKlSyv19ARBEFXJyy+/PMA575jsuIqJ+9KlS7Ft27ZKPT1BEERVwhg7XsxxZMsQBEHUICTuBEEQNQiJO0EQRA1C4k4QBFGDkLgTBEHUICTuBEEQNQiJO0EQRA1C4l4mtp8cwa5To5UeBkEQcxQS9zLx+V/uwT2P7a/0MAiCmKOQuJeJtMaR1fVKD4MgiDkKiXuZ4JyDtJ0giEpB4l4mNJ1D57zSwyAIYo4yqbgzxu5jjPUxxnYV+P27GWM7GGM7GWMvMMbWlX6Y1YfOQeJOEETFKCZyfwDApgl+fxTA1ZzztQD+CcC9JRhX1cM5h07aThBEhZi05S/n/BnG2NIJfv+C7cctABbPfFjVj87JliEIonKU2nO/A8CvSvyYVYlhy1R6FARBzFVKtlkHY+xaGOJ+5QTH3AngTgDo7u4u1VN7Ep1zcIrcCYKoECWJ3BljFwL4FoCbOeeDhY7jnN/LOd/AOd/Q0THpLlFVDaeEKkEQFWTG4s4Y6wbwMwB/wjk/MPMh1QY61bkTBFFBJrVlGGMPArgGQDtjrAfAZwD4AYBz/j8A7gbQBuC/GWMAkOWcbyjXgKsFqnMnCKKSFFMtc9skv/8AgA+UbEQ1AufGP4IgiEpAK1TLhM45NFJ3giAqBIl7maA6d4IgKgmJe5nQyZYhCKKCkLiXCU6RO0EQFYTEvUxQ4zCCICoJiXuZ0HSqcycIonKQuJcJaj9AEEQlIXEvE5wahxEEUUFI3MsElUISBFFJSNzLBIk7QRCVhMS9TFA/d4IgKgmJe5mgOneCICoJiXuZMEohSdwJgqgMJO5lgtoPEARRSUjcy4CobydbhiCISkHiXgaEG0OuDEEQlYLEvQzoFLkTBFFhSNzLgBB10naCICoFiXsZEKJOOzERBFEpSNzLANkyBEFUGhL3MqDpli1DnSEJgqgEVSfuW48O4fYHXsLpkUSlh1IQe5UMaTtBEJWg6sR9YDyF3+7rQzSZrfRQCmKP1smaIQiiEkwq7oyx+xhjfYyxXQV+zxhjX2WMHWKM7WCMrS/9MC1UhQEAsh7e5sgeuVOtO0EQlaCYyP0BAJsm+P31AFaZ/+4E8PWZD6swKjPEXfOwauoUuRMEUWEmFXfO+TMAhiY45GYA3+EGWwA0M8YWlGqAblS1usSdtJ0giEpQCs99EYCTtp97zPvKgk/xvrjbBZ1q3QmCqASzmlBljN3JGNvGGNvW398/rcewPHfviqZ94iFbhiCISlAKcT8FoMv282Lzvhw45/dyzjdwzjd0dHRM68mqzXPn3s37EgRRw5RC3DcDeK9ZNXMZgFHO+ZkSPG5efFXguXNHtYx3x0kQRO3im+wAxtiDAK4B0M4Y6wHwGQB+AOCc/w+ARwHcAOAQgDiA95drsACgKsZ85GVxp2oZgiAqzaTizjm/bZLfcwAfLtmIJsFXBZ471bkTBFFpqm6FqiI9d++a2c5SSFJ3giBmn6oTd8tzr/BAJsDZfqCCAyEIYs5SdeJeDe0H7BMPee4EQVSCqhP3aljEZBd0L4+TIIjaperEXXju3k6oUvsBgiAqS9WJO9W5EwRBTE7VibtaZbYMiTtBEJWg6sTdVxWLmPLfJgiCmC2qTtzVqvPcvTtOgiBql+oTd9X7i5iozp0giEpTdeJulUJWeCATQHXuBEFUmqoTdyuh6l11p4QqQRCVpvrEvco8dw/PQQRB1DBVJ+6KwsCYt6tlqM6dIIhKU3XiDhi+u5fFnWwZgiAqTVWKu+p5cc9/myAIYraoTnFnrGo8d6pzJwiiElSnuHs9ctepzp0giMpSleLuUxVvizslVAmCqDBVKe6qUj22DIk7QRCVoDrFnTFPL2Li1M+dIIgKU53i7vnI3brtZfuIIIjapSrF3acyR9LSa5AtQxBEpSlK3Bljmxhj+xljhxhjn8zz+27G2FOMsVcZYzsYYzeUfqgW1RS5k7YTBFEJJhV3xpgK4GsArgewBsBtjLE1rsP+HsCPOOcXA7gVwH+XeqB2DM/du6rpLIX07jgJgqhdioncNwI4xDk/wjlPA3gIwM2uYziARvN2E4DTpRtiLt6P3KnOnSCIylKMuC8CcNL2c495n53PAngPY6wHwKMA/jLfAzHG7mSMbWOMbevv75/GcA2877nbb3t3nARB1C6lSqjeBuABzvliADcA+C5jLOexOef3cs43cM43dHR0TPvJVEWpmsid2g8QBFEJihH3UwC6bD8vNu+zcweAHwEA5/z3AEIA2ksxwHx4vSskbbNHEESlKUbcXwKwijG2jDEWgJEw3ew65gSANwEAY+w8GOI+fd9lEozGYd5dxES2DJHMaNjZM1rpYRBzmEnFnXOeBXAXgMcA7IVRFbObMfY5xthN5mF/DeCDjLHXADwI4E95Gf0IVWGe3uHILuhevsIgyscj20/hbf/9PKLJTKWHQsxRfMUcxDl/FEai1H7f3bbbewBcUdqhFcanMqSy2mw93ZShOncimswiq3OksjoaKj0YYk5SlStUq6vlr3fHSZQP8b57+XNK1DbVKe5VtFmHh4dJlBFNF//TB4CoDNUp7l6P3CmhOuehyJ2oNFUp7j7V2+LOqc59ziM+nzS5E5WiKsVdVby+ExPZMnMd8fn08ueUqG2qU9wZPO652297d5xE+SBbhqg01SnuFLkTHkcEHxpN7kSFqEpx93r7AUcppIfHSZQPnWwZosJUpbirqtdLIe23vTtOonzIhKqHV1ITtU11irvHN8gmW4YQdgzZMkSlqE5x97ot42g/4N1xEuWDbBmi0lSluHvdc+e0QfacR6NqGaLCVKW4e99zJ1tmrkPtB4hKU53i7vUNsimhOufRaYUqUWGqUtx9Xt8gW+dQmHGbvttzE7JliEpTleKuKsawvVpDrnMOn8fHSJQXnRYxERWmKsXdpxphsVejd51bY6Qv99xERu4avf9EZahKcVeYKZyeFXcO1fRlPDpEosxoFLkTFaYqxd2niMjdmwuZODcmIIVRnftcRSRSyZYjKkVViruIih/f04v9Z6MVHk0uOjcSqgpjVC0xR6HInag0VSnuws/+25/swDeeOVzh0eRiiDszxb3SoyEqAdW5E5WmKsVdeO5ZnWMskQEA9EdT2PSVZ3ByKF7JoQEwfHbGGBijOue5irRl6P0nKkRVirvw3AFg1BT3Y4Mx7Dsbxd4zY5UalkTUuSuMUZ37HEVE7FmqliEqRFHizhjbxBjbzxg7xBj7ZIFjbmGM7WGM7WaM/aC0w3Si2sR9LJEFAGTM6+B4WivnUxeFZctQQm2uQpE7UWl8kx3AGFMBfA3AmwH0AHiJMbaZc77HdswqAJ8CcAXnfJgxNq9cAwYszx0AxpJG5C4ipVg6W86nLgqdGxMQee5zF2sP1QoPhJizFBO5bwRwiHN+hHOeBvAQgJtdx3wQwNc458MAwDnvK+0wnQjPHYD03MXlb8IjkTtjIM99DkPVMkSlKUbcFwE4afu5x7zPzmoAqxljzzPGtjDGNuV7IMbYnYyxbYyxbf39/dMbMSCX9gNALK0hq+lytWosVXlxF3XuqkKlkHMVqnMnKk2pEqo+AKsAXAPgNgDfZIw1uw/inN/LOd/AOd/Q0dEx7Seze+4AEE1mkZWeuxdsGapzn+vIhCqJO1EhihH3UwC6bD8vNu+z0wNgM+c8wzk/CuAADLEvCz6XuI8lM1bk7glxN4Sdkec+ZxFFMhS5E5WiGHF/CcAqxtgyxlgAwK0ANruO+TmMqB2MsXYYNs2REo7TgTtyH01kZCsCT1TL6IbnTu0H5i7UFZKoNJOKO+c8C+AuAI8B2AvgR5zz3YyxzzHGbjIPewzAIGNsD4CnAHyccz5YrkG7xX0skZUJ1bgHPHfHClWqlpiTWNUyJO5EZZi0FBIAOOePAnjUdd/dttscwMfMf2VnKrZMVtPxD4/swp9dtQJL2+tmY3iyK6RC1TJzFkqoEpWmKleo5kbuGVtC1Rm5nx1L4sGtJ/HMwfzVOfc/fxQPbj1R0vFZ7QfIc5+rUCkkUWmqWtw7GoIAnJG7W9yFXRNN5k+0/uTlHvz8VXd+eGZwUS2jUOQ+V6Ft9ohKU9XiPr8xBIW5PHe3LWN+ucZT+cU9kdaQyJTWpxfVMlQKOXfRyXMnKkxVirtYxNQY9qEx7DerZfIvYhJVNFGzTYGbWDpb8gobUeeuki0zZ5GRO03uRIWoSnEXkXtD0I/GkN+wZQosYhIR/XgBWyae0kreskDTObX8neOIKil3QlXTOV48UrZCMoKQVLW414d8aAr7jYSq+SVKZDTHF2oiW4Zzjlg6W3Jbxmg/IFr+krjPRQo1Dnv2YD/eee8WHO4fr8CoiLlEVYt7Q8iHxrAPY8mstF84B5JZS6xFRJ8voZrK6tB56VsWWKWQVOc+V7ESqs4PgPgcFrqSJIhSUZXi7pPi7kdD0Bm5A07ffaLIXXjtyYxe0npkoysk2TJzmUIrVMW+AxnqBUyUmaoUd8tz9yHoV5DRdMeON/ZIXHruecQ9ZruvlNaMbrNlKKE6N7Eid+f9QtTTJO41QX80hQ3//AT2na38DnBuqlLcG0N+BFQFi1vC8KsKMhqX9gvgjNwz5mVxvstge5VMKcWdi/YDCvWWmasIz919RZgxg40Mbb9XE5waSWBgPIWj/bFKDyWHqhT3pogfz33iWrzl/PnwqwpSWd1hyyQylpBrYhFTvsjdFuGXsmLGXudOpXBzk0ltmSxF7rWACCq92Nq5KsUdAOY1hqAoDAGV5dgyTs/dvAzO6khlnQJubzJWylp3aycmsmXmKlqB3jLkudcW4gos68HKiaoVd4Fhyzgjd4fnbrvfbc3YjytlxYyuG7aMSi1/5yziu+6O6IQYkOdeGwhR96LNVvXiHvAJcdchtlZ1RO62F92dVC2X5+7cINt7bzpRfgqtULUid/pcTMTb/vt5/PTlnkoPY1KEvnixzUTVi7uVUOVoDPkBAHGbUNsvf9217uXz3G3b7FGANufgnE+QUCVbphi2nxzB/t5opYcxKeJ9zHrw/ax6cQ/4jFNIZDQ0ho329HFbhG6fUXMi97J57qA69zmMXc/dEZ30aD0oBl5B0zk4N/JkXke8v168Eqt6cferhheTSGuoD/rREPThsd1nZfI0M4HnXq7Indsid9L2uYdd0N2Tu1XnTh+MQlTT1Y3QF0qolgG/apxCPKMhoDL869vX4pUTI/jir/cDADTbB6Q3msRwLC1/Lp/nbtW5U+Q+97C/57kJ1eoRrkpRTa8RlUKWESHuiXQWqsJw44ULcfXqDjx/aACA80X/9MO78L77t8qf4+ksIgHVvF2eOncS97mHPXLPsWWy5mV8FVgOlSJbRQu9stJm895Yq17cA6rlufvM28va63BiKA7Oec4HZP/ZqCxPjKc0tEQCxt+7SiGfOdCPH287Oa0x6bpV5+7B95woM/YKmRxbRq+eqLRSZGxrU7yOGKsXcyhVL+5+n+W5i4ZiS9siiKc1DIync7rypbI6+qMpAIbnXh/0IexXcyL3e585gi/8ej/SWR1ff/owklOwbaQtQ3XucxJ9oshd1rnT56IQ1bQWQEbuHrRlfJUewEyxbBkrcl/SVgcAODEUy3tpd3I4gXmNIcTTGiJBFZGAmuO5Hx+KYWA8ha8+eRD/9dQhBH0KfCqDT1Hwrku7JxyTqHNXyZaZkzgSqu7GYVmK3CcjW0Wee4Y89/JhT6iKyL27LQIAOD4Yh6YblSs3XrgAN164AADQMxwHYHSFrAv4EA6ojmqZjKbj9EgSAPCLHacBAI1hP+5+ZDf+7uGdBceSSGv47ObdGImnrfYD3v98EiXGbsu4FzFlyZaZFKu5mvdfo6zu3bEWJe6MsU2Msf2MsUOMsU9OcNzbGWOcMbahdEOcGFHnzrnV531xSxiMAccG48joOnyqgv9613rc88frAAAnhwxxj6c1RAJG5G63ZU4NJ2T0dWzQOFYkXgEglTW25rvpv57DKyeG5f0vHRvCAy8cw1gyK20ZnXNwzvGtZ49gYDxVxleC8Ar2Cd0d0aWrSLhKBeccj2w/VfQ5ywkw671o2I2slvGgzTapuDPGVABfA3A9gDUAbmOMrclzXAOAjwB4sdSDnAiRUAWsKD7oU7GwKYwTgzFkNQ6/KfrhgIr2+gB6hhMALHEP+1XHqtbjpvjbsX8wD/fFcHI4jh09o9h9alTef3okIW/b69x7hhP451/uxS9eO12isya8jCOhmlMtI5KF3hODcrHnzBg+8tB2PHdwoKjjhahXg+ee8bDnXkzkvhHAIc75Ec55GsBDAG7Oc9w/AfgCgGQJxzcpfpu4i008AKC7NYLjQ4YtY79/cUsEJ4dF5J5FJChsGata5sSg0ZvZHq3bZ+b9vWMYMuvlU7aM/ulR69Ttde6jiQwAoL/IyD2ezlIitoqZOKE692wZUYxQbLlxNVUUZau8WmYRAHtNYI95n4Qxth5AF+f8lxM9EGPsTsbYNsbYtv7+/ikPNh9ihSoA+Gy3l7RFcHIojoymOyaArtYITg4ZEXYspaEuoCIS8DkSqscH4wj6FKzvbpH3ZXUdC5pCAIB9Z6JyMZRD3G2Ru9F+wEioirYHokpnIk6NJLDm7sfw/RdPFPcCEJ5jwhWqHl7RWC7EVUrRtkwVWVderpaZcUKVMaYA+BKAv57sWM75vZzzDZzzDR0dHTN9agDOyN1ni9CbIn5Ek9mcyL2rJYzTIwlkNB2JjIawmVCNpzWMxA3BPj4UR3drBCs66uTfZTQuWwnsOxvF4CTibrdlRMOyYsT95eOGh//swdJMfsWw69QoxpKZWXu+WseRUCVbRtorxdatV1PnzGq3ZU4B6LL9vNi8T9AA4AIATzPGjgG4DMDm2UqqioQqAFkKCRi+eyqrI6NxxwTQ0RBEVufoM4U27Dc89yP9MVz0ucdxeiSBE4NxLGmL4HVLW+XfZc22woCxEMqK3K2I/4zNljFa/hpf9GhyYltm+8kR7DK9e2EJiXLOcpPVdLz96y/gW88cmZXnmwuQLeNETmhFnnNmipNBJdGq3JZ5CcAqxtgyxlgAwK0ANotfcs5HOeftnPOlnPOlALYAuIlzvq0sI3ZRKHIPym6RWYddE/YbProQ57BfwVjCiloHx9MYiqfR0RDEWy9cgNc+8wcAjBlazNJnx5I4M2YIufgAcs5dkbvVfmCyyP2PvvY8bvzP5wAARwYMcQ/51bzHlpqheBqprC6fl5g5InIP+JQ8pZDVYzmUiqlOaNVky2Tk+1mFkTvnPAvgLgCPAdgL4Eec892Msc8xxm4q9wAnw+G5K/bI3bgdS2kOWyZsJkmHTQsmHFDxnsuWYHFLGIARiacyGoI+FYwx+TgZXUdW09ESMXrG7zk9Zh5vfACHYoZIiqcy6tyNsjgRuQ+Mp2VUxznHL3ecyVn5erB3HACmtCJ2JgxEjdfh5HBikiOJYhHRekBVciL39BxcxDRVW6aa1gJYjcO8N9aiPHfO+aOc89Wc8xWc88+b993NOd+c59hrZitqB5ylkPYIXYhyPJ2F3yb6IiIejmfkz1et7sCXbrkIgCHWqayOoN/4G3E1kNU4MjpHt2mX7D1jinvGeFPFoqcVHfXyuVTGwG2Ru6ZzOam8enIEH/7BK/j604fl8ZrOccDcoGC2xH0wZlxN9OQp/ySmh/ie+1VWcLOOudR+YKqLktJV2DiMdmIqAw7P3RahBwpE7kLcRfJU/Cwmg2RGM8TdZ9yvSnHXkdF0LDVXv6ZcPuIp05JZu6gJgNE7XjE3yB6z9ZEXvvvguPH8P7I1JzsxFJePW8r+8hMhxjEYSyOWKt0+snMZYcX41VxbRloUVeAnl4qpeuhZzfnd8jIZD9tsVS/uhT13Q5xj6azDurE894zjZxGpiyg7ZP7MGINfZUhpOjgHuloicq9WAEiZEbZoaXCBKe7D8Yysc7fvACV8d1H7LpKwi1vCONw3Lo8rZX/5ibCvmu0ha6YkiCjOrypwf+eryU8uFVO9WrG/Rl5f71HVK1S9jkPcbbftkbv9finuOZG78b8QXfEzYHj5STOSDgdUdNQH5e9EdLHnzBg6GoJYOc+wZYz+MkbkHk1m0BgyerQJcRdXDvbzEDtDhf1qji0jWiY8tvssntrXN/kLUyQD49Y4xARFzAxR2x70KTldSdNzsFpmqnkG8Rpx7k27w061l0J6GntUruaplombm3gIwgHj/iFZLeO0ZcakuDu9fBFJ+1WGhc1h+Tvhue/sGcXaRU1orTP6ww/H07LlbzSZxXLTixfibq/QAYwvgPgSNIZ9SGasL8Khvije8MWnsPXoEL70mwP4+u8Oo1QMjKfkVcpJ8t1LgjNyL1QK6T0xKBdTTqjaJgGvv05yhWq1JlS9jLBNAKfQi8g7ntYc94dyInfRj8YU92SuuPtVRS6d9ikKFtnFPashns7icP84LljUhHYzqh+JZ2ylkBl0NgYRCahW5J7IIORX5MST0XT5JWgK+x22TO+Y8Tf7e6M4PZqQ1lEpGBxPYUVHPUJ+hSpmSoRIovp9LGezbPFzNfjJpSIz1RWqthfN668T7cRUZoQ1oyq5tgzgLJEsZMuI/8cShnAG/XZbhkmbxK8yLDLLJhkzPnx7To9B53BE7i11ASiMQdONyL0h5Ed7fVB63KOJDOY1hPDDOy/DW9ctNMRdRO4hvyOhKiaW/WfHEE1mMZ4q3WrSwZhR07+4JTKjyD2d1T1/CT1bOBKqttck44hIvS1apWSqCVV7tO7118nLi9JqStz9eUohAWeiVda5i4RqwGXLmJF7qEDk7lcVLDR7zLTXB5HK6Nhpri5du6gJAZ+Cr797Pb5z+0YwBtl+oCHkQ3PEjxHTjhmJZ9Ac8WPD0lbMawiai6SELeNHMmsXd2PC2XbMaE1Qysh9IJpCW10Q8xqCM2pJfMs3fo8vPb6/ZOOqZhy2DM8vVF6M9MpFZorVL9U0CYqrDC8GNlW/ExNgibuvUORut2V8+UshfaphkciEqi1y99s8d5+qYMOiJixsCmHFvHocH4xj9+kxtNcH0dloWDLXrzU2BQn6VCSzGjIaR0PIj+ZIQNbXjyYyaAr75VjtnntT2C8TuIBV877frIGPJo2ukcxetjMNOOcYiKXR3hBAIpPF/rPTb+h5uH9cXtHMdURC1b2ISUSkkYDqebuhlEw1iezw3D3egyfr4RxKTUTuAVO887UfMO63bisKQ8CnIGaKpz1CD/qUvJ67T1WkTeJXGS5Y1IQXPvUmLGwKI53VMRRLo7MxmCO25y1okG96Y8iHlohfTip2cferCtI2W6Y+6OxSKa4aRBCo6dyRcJ0u0VQW6ayO9rogmsJ+jCamd0UgrKfkLNXmex2hTQGfU9yFEEQCvqoo8ysVU7ZlHJ67tz9TVrWM9ybr2hB3U4jtEXqhyB2wfPeAqriajSmW5+6ydWTk7ro6SGU1Y7u+YO5F0MVdVsvghpAPzWE/RvJF7ub44mkNAZ+CcEB1iHe+PtjRZAa6zvHPv9gjvfJoMoOn9xdfJikWMLU3BNAY9mMskZGCk84WLz7jpk1UbL/uWseyZYz3VSRY01Lc1aoo8ysVVkK12Dp367Pv9e6ZmodtmZoQdyuhmlstAzgFGchduGT/Gytyt9syVuTubnGQzuqIpbOoC+Q2+upqDcsEq7BlxpIZZDUdownDc7ePP5bOIqgqCPkUJDKaFNd8q1XHklmcHk3gW88dxWZzh6c//Opz+NP7X5K9bADgxSOD+Ief78JgHj+912x+1lEfQlPYj7TZBnksmcH6f3ocTxU5UQgrKz5LC6+8jm5LqAJWgtVuy9h/rnWm3vK3ihKqOtkyZcVKqNqicH/+hCpgJVHDrs6LQVuHyJDfGfmLyD3geo5UVkc8peWN3BljuKirGQBkQpVzow+NpnOHLQMA4ykNfp+CkDk+0YogX0Q8nspKL/5If8xoVWxG8PbJ4LtbjuO7W47jhq8+6xB9ADhh7g/b3RpBc9iYhEYTGfSOJjGeysrfT4aYEMmWMbA3DrP/nLVF7oD3y/xKRbqWE6qa8731ErUh7r7cyL1QQzHASqKGXdF20KfIOmRHQlVRpJA6rB9VRVbnGEtmUJ9H3AHYxN2PloghoMfMnu1CUMX446ksAqoiJ53Pbt6N99+/FYmM5YWL3aCiyYy0bo4MjOP+54/KY+wbiOw5M4aAqqB3LIXjg3GcHIrbNiWJwacwLGwOyYlmNJGRkXjSfJztZpOzQpGXFblTbxogN3IXPwtxE4GA14WrVGSmuELVXknk9QlQJlTJlikPgbyLmHI3zhaIqDzkc4u7arvtnBzEZZejrbD5OMPxDCKB/OJ+/QXzsb67Gcs76tBk2jDHTXFvdHnu46ks/D4mJ5+tx4aw90wUibSG+Y0hqArDeQsaARgVMwlb5P67A9bOTeILEU9ncXQghrWLjX43qayGP71/K77w633mOOJY1BKGT1UscY9nZF5ArL79xWun8csdZ7DlyGDecxTinkh774v4/ReP45CtZ89soNkWMQH2Hu7G/2LynjPiPs2dmIzb3hNNOxny3MtLvkVMjDEZvatuW0YsXMoTuee7bU+65ptANJ2jPph/c41VnQ342YeuQKMjcjfsjnyeuz1yPzWcwFgyg3haQ3PEjwfefwk+8qZVAIwkprBfRhMZHOgdx/kLDeEXorzvbBScAxebVw9EW2CYAAAgAElEQVTJjI7heAZH+o3J5YS5naB9LCO2yF3sMrX3rNHe+PE9vXnPcUyKu7ci93RWx6cf3oUfvjS7+9FatozxPupS3F2Ru8eThaViqi1/7VGw17tnishd07nnqp9qStz9LhEP5qmiASxxD/vdEb09crduB+wbguRpTgYAkQK2jJ0WV+TuFvd4SkPAp8pxpLI64mkN0WQWIb+KN6zqwNJ2o5/8WDKT01zsipXtAGzNzMwNRS42N/pOZTUkMxpOjybMcRjbCQLIb8tkjIqZvWeM+von9vbm/QDbE6ri97rOceUXfovvbTk+6etSLsQq5MHx9CRHlhZpy5iRu+YS9znruU+rt4y3X6OsI/lL4l5y8lXLAMjZcEMgInb3VnZiMlBY4R2e8rUVBpA3oepGeOwics9NqGYRUJkjmQsYVS1CEIS3b7dlxLguMfd8FV+i3afH0BjyYbm50XcyoyOZ0XB2NImhWBqjiQyWtBq/ExbRWCIjV9Gmshr6oykMxdJYu6gJZ0aT2G1OGHZEQpVzy+/vi6bQM5zAtmNDOcc/ta8PP952EqPx8m7KLVbcDsRmV9xlnfuk1TLeFq5SMdWWvxmNy++A1yfAjK2+3WvWTE2Ie8CMkHwub118uQqVQuarlgEgt9gTFKqft1s3+Uoh3TSEfFAYcHQgBr/KZJmkGH8slTXq3F3jOmsTd1VhqAuoGE9lpf0CAOcvapJXAqmshg9+Zxse3HoC65e0yElsPJmFzo0vjxDdbjNybwj6wJgRhY9Jcdexx9xx6raN3QCQ178etXW4FFaRqNzJtzfrJ366Ax//yQ689/6tOb/72Ss9+PAPXnHc98DzR/H3P9+Zc+xkiM6fAwX2ri0Xmjuhar5NwmIQ+Zm50oJgqi1/M5ouXyOvRcNushp3bMXpJWpC3K32A+7IXTV/X8Bzz4nc89e/F9oQxC70xUTuisLQFPZD0zku7m6RzydtmYy1iMlOPK0hbEvYNoT8iCYzMnK//oL5uO2SLjmZxdMaHt/Tizev6cS/vWNdTt8cAPi9mRwVtowY20g8I6tpkhkN+84alszV53QAsGrj7YzZVraKWnch7kf7Yw4rh3OOoVgaAVXBaydHpAALPvaj1/DLHWccf/PY7l48sv10/hfVxqmRhGOiEY8tthKcLWRXSFfkLlYx1gXnli0z5Z2YdL0qks6cc2R1Lr+vXpusa0LcZYTuEvFAnkQrYJVAFrJl7BE54BR0Ry29I3Ivrk2PSKpevrwt5zE5N267xwUAEdt9DSEfxlOWLfPvt6zDrRu75WQjIu8NS1rQXh+Uj2cXvif39sGnMGnLADBbENgTqjoOnI1ifmMIi5rDqAuoOJtH3PNF7nLVbCrr2BAkmsoiq3M5Wbx6Yjjv62QXvp6ROKLJ7KQ2zm33bpGVQIC1EcngeNoxWRzsjZZVNNwrVDVNlEKa1TKBOVYKOdWEqsblBOjl10hUQYmqO6/VuteEuIs6cbf9IiJwd+Qu+snk2DI+y5ax4ytQM++M3Ce3ZQArifr6FbniDsBRLWPHHs3Xh3xGL5eM6I/j7mwptgp03j9iE8cTQ3Gs725xPG4+cR9NZKR91NkUQt9YbhRsvyJwiztg2FCCYTOavnp1B3wKw8vHLXG3XxUk01YVwhlz8/ETtsf89gvH8P0XrWTtaCKDE0Nxx1aBQ2bEntW5vLoYGE9h0388W9SVwHSx78QE2Dx3M3Ktm2Oeu4jYszrP2TA8H1lNlxNgsdF+JRCRusgPeG03ppoQ90KRu6yWKZhQdU8G+e8vlFydakIVAJojAQR9Ci7qbs4ZP2BMGHkj94A9cvdjzEyoBnwKFPP8xGQz6lplG3TdLxDVNQIh7iOyWkZDPK3J5+5sCBWM3EUlkGhPfGIoLjc1OTpg+fTCKlnUHMaahY0Ocd/RMypviwVRfdGk/NKctG0D+IMXT+CHL1mbix80O2ba2yzYq2QGTKE/NZyApvOy7jqluW0Z3WnLzNWEKlCcFZXRuLxS9bLnLjx28X2tSluGMbaJMbafMXaIMfbJPL//GGNsD2NsB2PsScbYktIPtTByJyZX5C7ETnUlWgsmVAtF7rbHLbRQqlhxv2VDFz7+lnOcpZY+Z+TunlwAl7gHfRhPZpDK6I5zEI9ptVCw2hn7bO2MBVeuanP8LMTdnlCNZzQZ3Xc2Bgt47hl0NhorZxM2z/3S5a0IqIojqSrKE1vqAljf3YLXekbkl39Hz4g8TlwBnLJF4nZBPjuWdIxlvxR3S9AHbX6+uF9MTuX04XMSqtxpy0RkVDozMXho6wkc7p/dBVrTYerirlfFBKhJm80ca7UlVBljKoCvAbgewBoAtzHG1rgOexXABs75hQB+AuCLpR7oRMhSyJzI3UxYFljElK/9gP1/6/GL8dyLs2U2XTAfH3jD8oKPb4/cW+sCEEMPuTz3qLmIyS7uAVfi1L3iVoh7QFVQF1Bx4WLr6gGAmVBNW7ZMRkMinbUid9OWcSdIxxJZzDfbIiTSGhJpDX3RFJa11WFJWwRH+y1xHzI3SWmNBHBRVzOSGR2H+8cRTWYci6REPx27zSIi92RGw2gig4HxNDSdYzSewcFeQ+QGY9b4hmJptJmWkiiL7DPFfSBavvJIa5s9Z+RuVcvMXLgymo5PPbwT3/195dYRFEtG4/LqtJhFSVmdI+RXoTBvi7uM3M3vWTWWQm4EcIhzfoRzngbwEICb7Qdwzp/inIuwaguAxaUd5sQUWsRUcIWq2H1pOtUyefZpBVCw/UAx2B/fryrmP4b2+oCsP484qmV8GDOrZexRvnuTb/vvQn5V3v9Xb1qJf/k/a3PaMnS3RjAcz8hLYbGISjx3Z0MIaU2XG44ARu18WtMx34zc+6IpfOXJAwCArtYI2uuDDq9feO4tdX6sMVfU7jk9hvf871Yc6hvHbRu7AFhXAKdGDHFf0VGHE0PGbRGxazrHL3acxvp/fhyP7jwDwBASkXMYHE9hVWe9vA3kRu5P7+/DW778TM6CsDOjCfzVg69iPDX1VbdWnXuhRUwzT6iOxDPgHOgZ9v6m5umsbkuQTi6AGU2HT2VynwOvknVH7h4bazHivgjASdvPPeZ9hbgDwK9mMqipIu2XAouYcnvL5LdlQv4Ctow9cs/TWyagKg5rZbrjt98O+VS01QXRGBLi7kx8JjM6xpIZR0QvJjORPHSuuFWkl/6GVR24+aLct9DuwTMmInfLlhHRuRDXnuE43nvfiwCAxeYuTN9+4Ri+8bsjeP2KNrx+ZRvCAdWx2GoonoZfZagP+rC8vQ4Bn4JHtp/GaydH8Pd/eB7evt6ICxK2yL21LoBz5jegx7Rlzo5adsyTe/ug6Rx90ZR8P4WQD8bSWDWvAYxZlTNis3Hx85N7+7C/N+pI1gLA7w8PYvNrp/H7w/n76UyEZkZ0uZ678X8pKkFE7qLH45uac86RttWtF5MgzWocPsX4Tnm5RUPW1SuoKj33YmGMvQfABgD3FPj9nYyxbYyxbf39/fkOmRYiYvW7BLZQ+4FC4i5EPTehaq1cVfJ0niy2UqYQfjWPuAdUtNYH0BAyvhTuqhbAECq7gCsKg19l1j6wtt/ZI3f3lYlgzYJGmRhtqwtYkbvf8twBI/rNajr+6sFXse9MFP9w4xq861IjzXJsMIaGoA8/+OBlmNcQQsivOMR9OJZGSyQAxhh8qoJzOhtk07Pr1nTK87RsGSMx29USQc9wArrOHUndrUeHINabbVxmrNAdjKWRyhptG+Y1BNESCchIXUxMwqbZfXpUPo8d4dHvtOUBBJNVfGicOz4rwoO377QF5O/TXyx2cfdaTxM7YkIT51ys5x7wGb2hvBYN28lJqBbpuY/aNsUpJ8WI+ykAXbafF5v3OWCMXQfg0wBu4pznzVZxzu/lnG/gnG/o6OiYznjz8tZ1C/Evb1sro1xBoEC1TLhAVUywUOSu5F8BK46fiSUD5Pf0P3rdKrzn0iV5I/dGKe7JnAkqoCo51TKA8Vq463LdKArD683ofV5DCImMhkTGqpaZ12BaL2NJ/GDrCbxyYgT//LYLcMeVy9BoTkI6BxY0h+Rjhvyqw/IYiqVlaSVgTCgAsLQtgsUtEflaijbHPcMJLGoOY2FzGGlNx2As7SjHPDuWxLrFzfj6u9fjL65ZAcCI3MUG6K31AbTVBXB0IIZUVpPiLnIWom+OOwIW1TWv2Sp4AGDf2TGc/5nHHMlfN5puXEWqzLkTU0bT4VMYmiNGLkUkfBNprWDHzUKIxPR4KpuTKPcSVrM0c+FWkZ67TzHsSS+Le04pZBGRO+ccr//XJ/Evj+4t69iA4sT9JQCrGGPLGGMBALcC2Gw/gDF2MYBvwBD24vd5KxGdjSG869LunPuFSLvr39d1NeHD167AZcvbXMcXWMSkWvaLHfFzoV7uxZIvSfvuS5fg8hVtaAwbj+22ZQBDKPOVc0rP3Z5QdUXxhbh6lTHpLmoJy+hZ1BzPMyP33rEU9p+NorUuIO0dn6rI12Nhs7VRdtgU93RWx2/39WLIjNwFwncXk4qYrBJpHadHEjg6EMNF3c1yQhiOp3F2LOl4j5a11+H6tQuwzGyqNjCellUkbXVBnL+wEc8fGsRN//k8zowm5Wu27fiQvKpwl0bKyP3UqCPK+vYLx5HIaI4STsD40n5vy3EMjKegc26Iu+L03MeSGdQFfVAVhta6oLx6+N6W47jtm1um1GvHvrLXy9aMsFWm0sM+kzU9dx/ztOcuzkXaMkUkVIfjGcTSGhY0lX8z+UnFnXOeBXAXgMcA7AXwI875bsbY5xhjN5mH3QOgHsCPGWPbGWObCzzcrJJvb1XAEP2Pv+XcnPLFoGsxkEBE1u7H8akKVIUhUkpbxjWBiMg97LfG2mwTR3fFT0C1bf5tF3TbOeUrtRT8n/WL8O3bN2LtoiZ5n5hYgj4V9UEfhuNpjCWzcpJxj8X+wQ37VSTSGn788knc/sA2bDs+7Ijc15ntiK8yJxXLlsniyb1G9cx153Va4h4zxH1hc1haSEvbDFEXk0bPcAL/8MguzG8M4fIVbfjSLRfh7hvXYH9vFNFkFufONyaUZ0w7KKAq6BlO4D+eOChXzArxHIql8YFvb8NPX+7BeCqLzduNi1Z3j50Xjw7h73++C9/43WFoOofKGBTmtGWO9MfkBNReH5C+/+7To+DcuRhsMpziXtqk6guHBtAXzS15nQ5yg5IpJJEzui4LC7xc5y7EfCoJVRFEdJmttstJUZ475/xRzvlqzvkKzvnnzfvu5pxvNm9fxznv5JxfZP67aeJHnB2sRUzFpRZk5O532zKFHyfoU2YcudujPPdqWmHB5PPcgdwoPFBAxIOO5GrhycinKrh6dYdjgnM/91gii7FERuYDBGISWNgUcvytsR+sdZxd3C/qasbmu67AW87vdDxGIq3h8b19WNoWwYqOOrmydzieRt9YEp2NQVlbv7Q9Is+9KezHAy8cxZH+GP7tHevQFPZDURhu29gtH1v0vf/dgX4EVAWXLGvBS8eG8OUnDuBvfvwaspqOwfGUHOeT+/rwk5d78JvdZxFLa2gK+3Pqyx9+xRD9X+w4A03nUBQmgwERuR8diGG5FHcrct9vlnFOpTJnKJaWdmEpI/dkRsN779uK+547VpLHkxVCU7FlNA6/anju6ax3t24U7QbEd7CYUsgTUtw9ELlXM9KWcQlmwePlDk1uWya/8BrPoTgsk+kiHjvgEt5C1TKCQv1x3L+zn5P7yiQf9r91ro41yjCjyUxOjkNcni6w2TIhvwqdWytXAWMBk50LFzfLLpx+c8HVUDyNLYcHcd15nWCM2WyZDM6OJdHZGEJHg2ETLWmz+uO01QeQzOi4bHkrrlxlVf+EAyquO8+YQIQVdKB3HBcubsLStjoZRR/uj+FH23owMJ7GlSvb8WdXLZd/LxK5V63uwGFb7X4yo+HRnWfQ0RDEmdEkth4dgqrYInedI57O4sxoUrZfNiL3FDKajsPmVYD9NfreluN4z7deRMwm+P/fr/bhjgdeMl+HNBY2h1Ef9JVU3A/1jSOrc7keYKZkXJH7ZDaLaMZlee7ejdwz0nMvvsxTrNXoavFI5F6tFEqoFsKqc3cKZiFbBjCSqW6Rmw7CmnGXVK6cV4/miN8h6I22iDknoVpAxMU52dsVTIT9b93J3NFEBmPJrMwHyLGIyN2VUAWcq0VbIxO/XuGAiiP9MaQ1HReY9pCwXIZiafSOpdDZGLIi9zbri9JeZwj+rZfk5mBuvaQLYb/qaNr29tctlpfIi5rDuGBRI3788kkMxdLobAziUzech43LWhFLZRFPaVCYEfn3R1MykfnC4QFEU1n8083nI+hTsOfMGFRmXY1lNS776yxrN+ru2+qDGBxP4/hgTAreeMqIUnf2jOKzm3fjuUMDuPuR3QCMCeT7W47jdwf6kc7qMjG9uCUs9+SdDtFkxhFN7ze7gPaPF7+C90j/uKN/kB3x2MICnSxyFwLpV43KL08nVHW3516MLZNAW12g6BXtM6Gmxd0qhZyiLZPjuYtFUrmP86Vb1uGuN66cyTABWF67++rghrXzse3T1zkiaZ9qWUFu/zxgOwd7T3pxTu6rkkLYn8/u9xu2jNGiwD2pWbaM03MHgCEzMr77xjW44cIFEz53JKBKH1lE7CG/irBfxaG+caSzOhY2hXDZ8jZctrzVkYOY1xhEY8iHTRfMz3nc169sx+5/fAuWd9QjEjAe78YLF8ga/avP6cCGJa3Yc3oMiYyGtnpjoqgLqIinNYynsqgL+rCywxDoux/ZhQe3npDbFl62vE2WYyoKk497ZGBcHmNF7kHE0xpeOWFV3cTNKP2Lj+1Da10A77t8CX76Sg8O9kbx+J5e2VHz6EBMivvrV7Tj2YMDODoQg6Zz3PPYvrx9c7YdG8pprzyayGDtZ3/j6JUv2jj05+mBf3Qg5ujdI/jET3fgLlcPfoGYuOqLXMQkBNKnKogEfHJBWjF85/fH8MZ/f3rWSkOtRUzFV8v0DMexeBb8dqDGxX3akXtOy9/8i6EA4NLlbQ5bYLqIx3Y/t6gHdyMi+UL9cdx2jVygNUGlTL7HAVyRe8gQ92gym+O5i+ec7/DcjccZjKXREPTh9iuXyZLKQoT9qlyZavfnW+sCsi59YXMYf/y6xXjozssdf/uJTefi+x+4rGBFkLhqOWd+A27ZsBgNIT9WdzYAAN58XidWdzbI3aTEc0eCPsTSWcTTWdQFfFgxzxD3R7afxgPPH8PJoTgagj40hf1yN6zxZBbt9UF0tYbxyvERKe4i+dtWbzz2C4cG5NhEInz/2SiuXt2Bt5kLuo4PxvHwq6fke7K/N4phU9z/4poVCKgKvvz4ARzojeJrTx3GL3acwbGBGJ450A/OOVJZDe/65ov41rNHHK/FF832yPbN1UXkPpBna8LbH3gJH/3h9pz7jw7EsPv0WN4JQYh5sdUyVuSuYGl7BMcGYkWL9VP7+nCkP+awqX76cg/+1VZ2+ONtJ0vWj0eci2z5W6Tn3tVSfr8dqHFxL7SIqRAdDUGct6ARaxY2Oe6fyJYpFWK/zWJXugpxz02oFqjhL3B/IewLndx+/2AsjURGyxu5t9cHXFG/tWq02KqicMCHpLnLlN2fb6nzyyoVe7mlna7WCNYubsr7Ozs/+fPX4zNvPR8AsLqzAc/+7bW49tx5OGd+vTym3RTguoCKeEpDLK2hLqiiqyWMhpAPAVXB0cEYjg0a0Rhj1laHosRyfXcLXjkxjCMD41jUHJbWVYd5VfDswQHZ/yaWMiaQvmgKS9vrZGL6zKixXeFN6xbCpzDsPzuGobgh7h0NQbzr0m78cucZWZ55eiSBL/x6H95731b85YOv4vhgHGnNKC0V9I0l8f0XTxivmc3/PWBG7kOxFDSdI53V8cj2U8hqOk4MxfHswQEc6ovK42O2fv3PHbImiaf29eFQXzTXcxftfwuIvDjerzKs6KjHaCLjsPQKwTnHzlPGrmFigxkA2Pzaafxgq3GeyYyGv/3pjpxJbrqIBKrVFXLiiUvTOU6PJGalUgaocXG/enUH/uKaFfIyejLCARW/+sgb8LolLY77ReRcrL0zHaT1U+RzFBL3QpG7ZcsUJ7D24+ybfzeGfTKybXSVQv7hhQvxvsuXOh/HHMfAeLroDU3sk0mrzXJpiQQggqNFBcS9WFSFOXIP4gu3yoziAaNG3hiPEbnHTFvGpyp4/P9ejbvfugbprI6Xjw+j26x+uLjb2Yzt4q5m9EVT+O2+Ppy3wHrsdlPcB2NpXHPOPABALJ3FcXN/3SVtRl8en8Kw72wUY8ksVnXWY1l7HbafHEEyo8s8xJUr26HpHD/eZnQJOT2SwLHBOBRmVO88YZaU2n307ScNO6izMShzB6OJDM6MJrGoOQydG/mNJ/f24iMPbccTe3ulmH3H1qzMHiU/e8C4CvnRSyfx/gdewqcf3mX1sLetUP2v3x7EFV/4LU4M5tpHwtrwKQpWmN/bw3m2dgSMxWH//2unkc7q6IumrOqjs9Y+v2dGE8ZGL2a/f86d4j8TMi5xz0wSuZ8dSyKjcXSTuM+c5kgAn9h07oxFWTQkczcmKyXCc3fXuReikC0TKCDicuOO6UTufmfkLnDbMjetW4i/fNOqvM87GJtC5G5rD2EvwxRiFvarsjSy1DSG/HLiENZJXdDw3GMpq0Pm/KYQzplviPV4KiujX/ekenF3izzmI29aLe8Xjw0A157bAYUB8ZSG44OWfaMoDJ2NIbwk9rttjWD1/Aa8eMT4WUT8F5lrBcRq2lMjCZwYjMlFer94zWiq1jeWQs9wHI/v6cWuU6NQFYbLl7dZiWHTIrpqtbHmYGA8hT7TatliPmdjyIcn91rrFIW/v7QtgucODWA0kcHfPbwTIb+CbceH5ebk4r0fjqXxjWeOoHcshTu/uw2prIbB8ZSsCnJE7qb9Za9MsvO7g/34ywdfxSPbT2Gnee4Kc4q32OilZzguJ879Z6NFbRoyGVYppGk9jqcmXIgmJqmlJbBxi6Gmxb1UWJF7GcW9QH+cQhSM3FURuefvs1Ns5G6vhQ+7PPd8twshhDqZ0Ytu0yCer9VVMikWLS1sDjmSxaVmtdlJ0h65a7qx96t9TYOoWQeci1J+8IFL8eV3rgMAnLegEc0RP95z6RKHXWQX98uWt6EuaGydeMwUILFx+cLmEA6YdfCLWyI4t7NBertixXBLXcAxlsP944ilNVx7zjwEVEVuct4XTeGbzxzBnd/dht/s6cWqefXobAzJhnL3v3AMi1vCuGndQgBGUlVYIi8eNcR9XVcz+settsqibvtN53WiL5rCkX6jlPLdly6BpnO5EE1ctX3vxeOIJrO448pl2Hc2iu0nRnDbN7fgUz8zkrri3PyqggWNIYT9akGP/Kl9fXJsO0+NQmHA5SvapLiPJTOIpqw2FmLijKc1R/voD3z7pYLbPQJGm4+n9+cuvHd3hfzKEwdxzb89Ja+K3Ij3QbTcKDck7kVg1bmX35YpOnKPiMVNBfrj5CRUVcf/kyEmB4U5k6t2K8Zty+TDPjEU2/M+UkjczZ8L+e2lYsPSVoc/LsbdH005JqjWuoCcZO2X2q9f2Y63XWwkQwM+BU//zTX47E3nO54j6FPREPLhnM4GtNcHURfwIZ7O4vhgDG11ATlxzrdVHnW3RXDrxm58YtO5+J/3vA5vWGX1ZxJXCJ2NQZmUXN5Rh9W2HMJoIoP9vVFpTaxd1ITGsF9aS1uPDuF9ly+VCfGB8ZSsjtlnWh0XLGpCOqvj2GAcm77yDH6z5ywiARXnmHaW6NXz5jXGquJf7TprvIZm5N47lsKly1rx3suNRnOH+sdxqG8cj+/pRTydlZG7TzVss+UddXnFnXOO35rivvXoEF45MYwVHfVY392CowMx7Ds75tjoxRB3ywbaawrti0eH8MTePpl/yMeXfnMAH/zOtpxFShlXKSQAjCQyeN99W3NaSANGa+tFzWH53S03JO5F4J+gWqZkz2FOIMUsMAJskbsrEg+ok3juxdoy5uNGAj5HlDyRLZMP+we/6Mjd/Bv3Yich9jP12yfjz69egSc+drX8WeQcxpJZR30yYwwrzNLGiVYcNkcCOe2oAWDT+fNxq9m/PhJUEUtpODoQwxJb3b5IqjaF/WgM+dHREMRfXLMCmy6Y73hM4fW/8dx58r7u1gjOX2BcLYjPhT2qvHBxk7S3frnDsG7e/rrFMpFsiLsRuXNufIZWmVbJ0/v7sO9sFFuODKG7NYIO8ypizxnDHulsDOGN586TCVT75+Dzb1uLhc1hqArDC4cGoXMjAf3XP3oN7/zG7wFYFWorOupxsHccWU3Hk3t7ZSfNQ33j6BlOYNW8epnovf6C+Th/YSM0nWPTV57FV544IJ/z1HACxwZjWDWvHgqzJiHRguLp/X042BvFi2YDNyHknHM8e7AfGY3nlJJuPzGCoE9xfE4/dM0KjCYyjgokwZ4zY3IB3WxA4l4EInIvtqRyOkw7oeruLVOgnr1Qx8tCiMnA3bvGvnCpmMi90ErXiRDP2eYSd1HPXu7IXVWY64rDZ7vtPIflZtJv8TRWHN7zjnV4/xXLABjN50RC1e7JLjDFfbLl6jdftBCf2HSuvGIQYxJisq7LEPlkRpdXGRd3t8jP0f7eMdQHfWitC6A+6EPQp5i2jJWEXdgclongnadGHc8zz1wtvPu0ERF3NARx20ZrIVnAp2DT+fPxhbevxcp59fCrChY2h/CcrRT0V7vOoqs1gj9Y0yknq9evaMOpkQSuvudp3PHtbfjhSyfwuwP9uPXeLfApDB9/yzny9bv9ymW47rxOfOu9G9DZGMQTZm6gMeRDz3AcJ4biOGd+A5a218krkWcO9CPoUzAwnsabv/wM3nnvFpwaSWDF3z2K7205jmODcZwedbaJ/vYLx/CfTx7Ew6+ewq2XdKHBNqs8HnEAAA7tSURBVOF/+NqVaI748audZ3BiMC69/URaw5H+8VmzZACg/MukaoCpCu90kAnVIiN3sfTe3bzLKnl02TIFdpkqhLB13IJcaKVsIexXCsWuyhPPae8eCViVM+UW95zx2BLBEdc53LaxC10tkaLtroLPEVAxYrZWsPv3wpaZrMKiIeTHX1yzQm5kMq8hiHBAxYWmz3/lyg68dMzwle+6diXOX9SI8xc2ydbB+89GpR3DGENHQxAD42lHGeL8xpAU9102cV/SFpGfx31nogj7VdQFVKy3VQ75VQX/8yevc4y5uzWC5w8ZkfK7Lu3G3jNj+M7tG9Fgy+W885IuDIyn8NXfHoKqMBwfiuOZgwPwqQwP3XkZLu5uQXdrBO+8pEtO/tet6cQvd57Bw6+egqowXNzdgmODRv37jRcuQDKj4/hgHKdHEjjYN44PXbMC//O7w7IS64hpA/39z3fhdnPyBQxxT2Y0fP7RvUhnjfbNH7xquePKNhLw4Q/WdOKnr5zCz7efxt03rsHtVy7D/t4odA6K3L3GrNS5T1Hc33TuPHzvjktluZggUMB+kX1zpriIyV2NI7xghaGo0sZpRe7m39iTjgBw7oIGnL+wEZcsbcn3Z2XDfp71roqf1y1pxUeuW+X+k2k9x7HBGDi3onXAauVQbC+SjgajfFJYOxd3t+Anf3453nmJtSXD0vY6nG+u5WgOCwsmLbdKFI/TF01icDwtJ/EFTSG0NxjHH+obx7yGIP73fRvwgTcsQ1tdEAoz7JWOhiAYY2CM4fNvuwBNYX9OEAIA3a3GFUp90IfP/9EFePhDVziEHTAmmrveuAq7//EtWNZeh9MjCZwciuOirmZsWNoKVWH43cevwYfMXv6CS82VwvMbQ1jSFsGB3nFoOseStjosag7h1EgCr5hJ1BvWLsAdVy6Tr7uosAGA+54/Kr8L/dEUXjk+jHRWxweuXIYvv/MiecX21nUL8ZV3XgQAuGVDF0I+BS0RPza/dhoA8IjZTfT8WRR3ityLQFbLFNldcjqIKplirw58quJojCWw2g/kj9yLr5YxHsctyJGACp/CEAmoRfWoEfvBZjRedOQu+se7I/f2+iB++VdvKOoxSon9NZjpxiyFqAv65F6znTZx726NIORXio74VIVhdWeD4/J/w9JWZDUdjBneub0Xj110O23ivqytDs8c7MdoIoOrVnfgmQP9mN8UQmskAMbMTVmaQniT2YwNgOxRL6J4wNiX4N3mLl1uxAS0tD0yafWTX1WwqDmMnuEEeoYTslwTQN6/FW0gFjSF5MTY1RrGNas7MBxLI5rMYpe56GlZex0+/YdrsKKjHp/82U6ZwP31R9+Ap/f3Y35jCB/94XYMjKdwdCAGVWH4yHWrHBPRf952sby9YWkrdv3jW/DfTx/GPY/txz2P7cP9zx/Duy/tnpZ9N11I3ItA1LcHfOWM3KeWUC1EocSpVUVT3OMzxsyOl76c+xvD/il1wgz5VWS0bNF/Y1XLzE5VwWTYyx9n2t65EPatGu2Re3MkgOc/8caciW4iHvqzy3Kqrnyqgra6IGKprEN87eI+v8m6/5z5DfjZq0a0uXFpC7YcGcSqznr4VAWtkQAGY+mcDSfmNRji3l5f3FiF1VRs+46FzWH8/vAg0po+6RL+Ze11WNAUwtL2OtxySReaIn689cKFCAdUaev9/sigUalkvqciMXq4fxyMAavmNeDc+Y3gnOMTP92BgfE0Xjk+jAsWNeVcYbhhjOH6C+bjnsf242tPHcZ153XiH10VU+WGxL0IZiNyD5TI1w8UXKE6tcjd+BslJ6EKGILgtmsmIuxXEU1mi16hKh67tS44yZGzg8NzL0F757zPYXtt7PYIANnArFgKrT/oaAiivT7giHQbQj4Z0duf91xb5L+iox7P/u210m/vaAhiMJZ29BAS9+MMHJPHRAhxt19JTMTilrBsRDbZEn7GDE++3uz5c8sGy5YS4r7r1KhcAAZY1ViH+sbRHPbLaiSRgzgxGMf2kyP4oNkGejKWd9Tj2nM60BDy4553XFjWFe75IHEvAt8see72TTumS6FSyJA/f0Q/EUG/mlfMpiruckPyIoXxgkWNOHd+gyy7qzSOapmyRe5Wl898/nQpuP2KpTkWhqIwNIaMNs72mvpz59vaMNQHHZaNIfJRxxUGYIl6R/3EjeEEKzrqce78BlyxItdezIe9lXQx/VkKXRGIUlpN51hiexxxdXRiKC53zBK01wfx3KEBZHWODUuKz/nc//6NRR9bakjci2A26tyDPmXGlgxgVbm4H6tQFc1EvGtjt6MfiuAfbjxvSlcxYiKoK7L9wMp5Dfj1R68q+vHLjX0iK5u4i7YGjeVbffuODV15728ye/TbI/d5DUE0R/wYiWdyEtvCdlnQnGvLAMVH7uGAOqX3eVFzxHZ7+hVT8xqCMg/UbbtqEJG7znMX0LXXB+UagXVdzv5BXoXEvQhmo8791o3dcmOKmVAocu9oMDaKnkq2/v++eXXe+1+3pHVKYxK1+OVKRpYbxUwgx9Na0atsp4oosXRbHbOBuFLotHnujDGc09mAF48OyQ1QBMKeKRi5FynuU0VE7jPd7EJRGOY3hXByKOFYMNYU9kuLyi3u4pwW2Wr9vU51fttmGZ/C0F4fzIlUSsnKefVYWQIbonA/d7UilSYAEDatoGI9dy8SCfgMcS9T5C5KLN1++2zQFPYbn3GXiJ+/sAmv9Yzk7LjVbgqde6zi587G8ojf/MYQVIWVZLOLhU1hnBxKONYPqApDc9iP4XgmV9zrnU3aqoHq/bbNIowZtbQzXagyG0y1zcBsIGyNYrtCepG6oIqB8fJNUOKqZn7T7C7QAoz1BPObQjmlrXe9cSVuXLcgxya6ad1C6JzLnaYEbzqvE/9x60VYW4Ir0Hz4VAXdrRHZ8mEmCFtH1NoLWuoCecVdTGgXFrFXgFcgcS+S2djzsBRYm3x7R0jFpFjtkTtQvglKlFjOL1PUOxEfe/PqnL4pgGFNuEUOMKpNPnRN7taSAZ+Cmy9aVJYxCv73fRuKansxGRcvacH2kyM5ZZutkQCOIJZTeiquSkSDtmqger9tRF7WLmrGX71xJS5f0Tb5wbNETUTuARUBn1K2pLpo4DWbi1wES9rqSrJV5GywvMiNdybjTy5bgj+5LHdxlah1dyeR33juPHzrvRtmfXX0TCjqk8oY28QY288YO8QY+2Se3wcZYz80f/8iY2xpqQdKFEfAp+Bjf3COp640ZEK1CmytQkSCvrIlUwGjx/f9778E19q6OhKzj+hf5I7cfaqC69Z0lnUfgVIzqbgzxlQAXwNwPYA1AG5jjK1xHXYHgGHO+UoAXwbwhVIPlKheGoI+o23BLC/iKCV1AbWs1T6MMVx7zrwZr3MgZoaM3D2ygG4mFPNp3QjgEOf8CAAwxh4CcDOAPbZjbgbwWfP2TwD8F2OM8WK3LSdqmj+9YqljY4lq5N2XLsG15yQmP5CoakTLixaPtL6YCcWI+yIAJ20/9wC4tNAxnPMsY2wUQBuAARBzngVN4Zw+JNVGviZtRO1x/QULMJ7Syr4hzGwwq9fJjLE7GWPbGGPb+vtzdyohCIKoJF2tEXzszaurylsvRDHifgqAfd3yYvO+vMcwxnwAmgAMuh+Ic34v53wD53xDR0d1X6YTBEF4mWLE/SUAqxhjyxhjAQC3AtjsOmYzgPeZt/8YwG/JbycIgqgck3rupod+F4DHAKgA7uOc72aMfQ7ANs75ZgD/C+C7jLFDAIZgTAAEQRBEhSiqtotz/iiAR1333W27nQTwjtIOjSAIgpgu1Vt4TBAEQRSExJ0gCKIGIXEnCIKoQUjcCYIgahBWqYpFxlg/gOPT/PN2zM3Vr3PxvOmc5wZ0zsWzhHM+6UKhion7TGCMbeOcb6j0OGabuXjedM5zAzrn0kO2DEEQRA1C4k4QBFGDVKu431vpAVSIuXjedM5zAzrnElOVnjtBEAQxMdUauRMEQRATUHXiPtl+rrUCY+wYY2wnY2w7Y2ybeV8rY+xxxthB8//q2a03D4yx+xhjfYyxXbb78p4jM/iq+b7vYIytr9zIp0+Bc/4sY+yU+V5vZ4zdYPvdp8xz3s8Ye0tlRj0zGGNdjLGnGGN7GGO7GWMfMe+v2fd6gnOevfeac141/2B0pTwMYDmAAIDXAKyp9LjKdK7HALS77vsigE+atz8J4AuVHucMz/EqAOsB7JrsHAHcAOBXABiAywC8WOnxl/CcPwvgb/Icu8b8jAcBLDM/+2qlz2Ea57wAwHrzdgOAA+a51ex7PcE5z9p7XW2Ru9zPlXOeBiD2c50r3Azg2+btbwP4owqOZcZwzp+B0SLaTqFzvBnAd7jBFgDNjLEFszPS0lHgnAtxM4CHOOcpzvlRAIdgfAeqCs75Gc75K+btKIC9MLbmrNn3eoJzLkTJ3+tqE/d8+7lO9IJVMxzAbxhjLzPG7jTv6+ScnzFvnwXQWZmhlZVC51jr7/1dpgVxn81uq7lzZowtBXAxgBcxR95r1zkDs/ReV5u4zyWu5JyvB3A9gA8zxq6y/5Ib13I1Xeo0F87R5OsAVgC4CMAZAP9e2eGUB8ZYPYCfAvgo53zM/rtafa/znPOsvdfVJu7F7OdaE3DOT5n/9wF4GMYlWq+4PDX/76vcCMtGoXOs2feec97LOdc45zqAb8K6HK+Zc2aM+WGI3Pc55z8z767p9zrfOc/me11t4l7Mfq5VD2OsjjHWIG4D+AMAu+Dcq/Z9AB6pzAjLSqFz3AzgvWYlxWUARm2X9FWNy09+G4z3GjDO+VbGWJAxtgzAKgBbZ3t8M4UxxmBsxbmXc/4l269q9r0udM6z+l5XOqs8jSz0DTAyz4cBfLrS4ynTOS6HkTl/DcBucZ4A2gA8CeAggCcAtFZ6rDM8zwdhXJpmYHiMdxQ6RxiVE18z3/edADZUevwlPOfvmue0w/ySL7Ad/2nznPcDuL7S45/mOV8Jw3LZAWC7+e+GWn6vJzjnWXuvaYUqQRBEDVJttgxBEARRBCTuBEEQNQiJO0EQRA1C4k4QBFGDkLgTBEHUICTuBEEQNQiJO0EQRA1C4k4QBFGD/D/ip9bmAldh9wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        #np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        N = x.shape[0]\n",
    "        gradient_w = 1. / N * np.sum((z-y) * x, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = 1. / N * np.sum(z-y)\n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, gradient_w, gradient_b, eta = 0.01):\n",
    "        self.w = self.w - eta * gradient_w\n",
    "        self.b = self.b - eta * gradient_b\n",
    "            \n",
    "                \n",
    "    def train(self, training_data, num_epochs, batch_size=10, eta=0.01):\n",
    "        n = len(training_data)\n",
    "        losses = []\n",
    "        for epoch_id in range(num_epochs):\n",
    "            # 在每轮迭代开始之前，将训练数据的顺序随机打乱\n",
    "            # 然后再按每次取batch_size条数据的方式取出\n",
    "            np.random.shuffle(training_data)\n",
    "            # 将训练数据进行拆分，每个mini_batch包含batch_size条的数据\n",
    "            mini_batches = [training_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "            for iter_id, mini_batch in enumerate(mini_batches):\n",
    "                #print(self.w.shape)\n",
    "                #print(self.b)\n",
    "                x = mini_batch[:, :-1]\n",
    "                y = mini_batch[:, -1:]\n",
    "                a = self.forward(x)\n",
    "                loss = self.loss(a, y)\n",
    "                gradient_w, gradient_b = self.gradient(x, y)\n",
    "                self.update(gradient_w, gradient_b, eta)\n",
    "                losses.append(loss)\n",
    "                print('Epoch {:3d} / iter {:3d}, loss = {:.4f}'.\n",
    "                                 format(epoch_id, iter_id, loss))\n",
    "        \n",
    "        return losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "# 启动训练\n",
    "losses = net.train(train_data, num_epochs=50, batch_size=100, eta=0.1)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(len(losses))\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "观察上述Loss的变化，随机梯度下降加快了训练过程，但由于每次仅基于少量样本更新参数和计算损失，所以损失下降曲线会出现震荡。\n",
    "\n",
    "------\n",
    "**说明：**\n",
    "\n",
    "由于房价预测的数据量过少，所以难以感受到随机梯度下降带来的性能提升。\n",
    "\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结\n",
    "\n",
    "本节我们详细介绍了如何使用Numpy实现梯度下降算法，构建并训练了一个简单的线性模型实现波士顿房价预测，可以总结出，使用神经网络建模房价预测有三个要点：\n",
    "\n",
    "* 构建网络，初始化参数$w$和$b$，定义预测和损失函数的计算方法。\n",
    "* 随机选择初始点，建立梯度的计算方法和参数更新方式。\n",
    "* 从总的数据集中抽取部分数据作为一个mini_batch，计算梯度并更新参数，不断迭代直到损失函数几乎不再下降。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 作业1-2\n",
    "\n",
    "1. 样本归一化：预测时的样本数据同样也需要归一化，但使用训练样本的均值和极值计算，这是为什么？\n",
    "\n",
    "2. 当部分参数的梯度计算为0（接近0）时，可能是什么情况？是否意味着完成训练？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 作业 1-3\n",
    "\n",
    "1. 随机梯度下降的batchsize设置成多少合适？过小有什么问题？过大有什么问题？提示：过大以整个样本集合为例，过小以单个样本为例来思考。\n",
    "1. 一次训练使用的配置：5个epoch，1000个样本，batchsize=20，最内层循环执行多少轮？\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 作业1-4\n",
    "\n",
    "#### 基本知识\n",
    "\n",
    "**1. 求导的链式法则**\n",
    "\n",
    "链式法则是微积分中的求导法则，用于求一个复合函数的导数，是在微积分的求导运算中一种常用的方法。复合函数的导数将是构成复合这有限个函数在相应点的导数的乘积，就像锁链一样一环套一环，故称链式法则。如 **图9** 所示，如果求最终输出对内层输入（第一层）的梯度，等于外层梯度（第二层）乘以本层函数的梯度。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/2beffa3f3d7c402685671b0825561a91c17216fe8b924f64b9f29a96f45cbc85\" width=\"200\" hegiht=\"\" ></center>\n",
    "<center><br>图9：求导的链式法则</br></center>\n",
    "<br></br>\n",
    "\n",
    "**2. 计算图的概念**\n",
    "\n",
    "（1）为何是反向计算梯度？即梯度是由网络后端向前端计算。当前层的梯度要依据处于网络中后一层的梯度来计算，所以只有先算后一层的梯度才能计算本层的梯度。     \n",
    "\n",
    "（2）案例：购买苹果产生消费的计算图。假设一家商店9折促销苹果，每个的单价100元。计算一个顾客总消费的结构如 **图10** 所示。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/46c43ead4fa942f5be87f25538a046ff9456516816274cbcb5f6df3768c0fd34\" width=\"400\" hegiht=\"40\" ></center>\n",
    "<center><br>图10：购买苹果所产生的消费计算图</br></center>\n",
    "<br></br>\n",
    "\n",
    "*  前向计算过程：以黑色箭头表示，顾客购买了2个苹果，再加上九折的折扣，一共消费100\\*2\\*0.9=180元。\n",
    "*  后向传播过程：以红色箭头表示，根据链式法则，本层的梯度计算 * 后一层传递过来的梯度，所以需从后向前计算。\n",
    " \n",
    "最后一层的输出对自身的求导为1。导数第二层根据 **图11** 所示的乘法求导的公式，分别为0.9\\*1和200\\*1。同样的，第三层为100 * 0.9=90，2 * 0.9=1.8。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/c251a2c290e946f99ce3a3381396c392b50e5a4243c346509bd91177b7f2da90\" width=\"200\"  ></center>\n",
    "<center><br>图11：乘法求导的公式</br></center>\n",
    "<br></br>\n",
    "\n",
    "#### 作业题\n",
    "\n",
    "1. 根据 **图12** 所示的乘法和加法的导数公式，完成 **图13** 购买苹果和橘子的梯度传播的题目。\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/4ce8715f03f9477699707056544b1e6363f78aa09fda411d972878abb6d1d26f\" width=\"300\"  ></center>\n",
    "<center><br>图12：乘法和加法的导数公式</br></center>\n",
    "<br></br>\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/2fc6665e10f34f9e863172bb399862319f0914467d72457d9e7328616bdbe6df\" width=\"500\"  ></center>\n",
    "<center><br>图13：购买苹果和橘子产生消费的计算图</br></center>\n",
    "<br></br>  \n",
    "\n",
    "2. 挑战题：用代码实现两层的神经网络的梯度传播，中间层的尺寸为13【房价预测案例】（教案当前的版本为一层的神经网络），如 **图14** 所示。\n",
    "\n",
    "<center><img src=\"https://ai-studio-static-online.cdn.bcebos.com/580f2553aa4643809006f5a8d3deb2aa8dd4e1aa69d94cf6a35ead5fe7cf469e\" width=\"300\"  ></center>\n",
    "<center><br>图14：两层的神经网络</br></center>\n",
    "<br></br>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
